[Interest] Mouse/Pen input latency in Qt Quick and render loop

[Daljit Singh]

Hi Andreas,
the latency issue is platform independent (I have tested my application on Windows, Linux X11 and Android.
When V-Sync is enabled, there is a noticeable delay on displaying the rendered frames. For example, on a 60hz monitor the delay can be up to 16ms. When VSync is turned off this doesn't happen because the rendered frames are displayed as soon as possible without being limited by VSync throttling. As mentioned here https://doc.qt.io/qt-5/qtquick-visualcanvas-scenegraph.html#scene-graph-and-rendering:

"NOTE: The threaded and windows render loops rely on the graphics API implementation for throttling, for example, by requesting a swap interval of 1 in case of OpenGL. ...Without blocking in the swap buffers operation (or elsewhere), the render loop will run animations too fast and spin the CPU at 100%. ..."

Indeed if VSync is disabled (setting the swap interval to 0), the QML animations go crazy fast. In order to avoid this it's possible to use the "basic" renderloop, the only problem is that the basic render loop makes the application single-threaded (as there is no dedicated render thread) so if rendering a frame takes a little bit (like >5ms) then QML animations appear jerky as while the rendering is ongoing the animation events are blocked.

Now I think there might be some ways this could be solved:

- the vsync issue is commonly known in competitive gaming where input lag is quite important, one way to solve this would to limit the framerate of the application to be slightly lower than the display refresh rate (this https://medium.com/@petrakeas/vsync-with-low-input-lag-50ms-lower-2437118bfa5 provides more details). However, Qt provides no way to limit the framerate of an app.

- the possibility of dynamically turning off V-Sync at runtime would allow an app to operate in "low-latency" mode when required. Currently this is not possible in Qt, as the QSurfaceFormat::setSwapInterval needs to be called before the QML scene is created.

- make the threaded renderloop not rely on Vsync throttling so low latency and multithreaded rendering is possible, this was what I wanted to know in my previous reply. There seem to be changes to the render loop in Qt 6, but I am not sure if this has been implemented.

Best,
Daljit

________________________________
From: Interest <interest-bounces at qt-project.org> on behalf of Daljit Singh <daljit97 at live.com>
Sent: 15 August 2020 01:08
To: interest at qt-project.org <interest at qt-project.org>
Subject: [Interest] Mouse/Pen input latency in Qt Quick and render loop

Hi, I am developing a cross platform application for displaying pen input. The project requires real-time rendering of digital ink (stylus input) and hence the latency of the rendered strokes is very important. To render the strokes I am using a custom QQuickItem together with QSGRenderNode. Unfortunately, drawing the ink results in significant latency as the input is rendered with some delay (e.g. comparing it to something like OneNote I would say there seems to be around 10-15ms more delay). I came across a previous mail thread here https://lists.qt-project.org/pipermail/interest/2014-March/011692.html where it is mentioned that this is largely due to VSync and it's possible to disable this by calling QSurfaceFormat::setSwapInterval(0). This works well and reduces the latency significantly, unfortunately it also introduces another problem: the render loop for the Qt Quick scenegraph must be set to "basic" otherwise QML animations go a bit crazy.

My application requires that some of the ink to be rendered when the user is scrolling a large canvas (for which I am currently using Flickable), but using the basic render loop results in the animation being quite jerky (so scrolling is not smooth at all) as while the strokes are rendered the animation system is stopped (at least this is my understanding). Is there a way to overcome what I described? I only require the app to be in "low-latency" mode while the user is drawing on the screen and for all other scenario I would like to stick with the threaded render loop.


Best,

Daljit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/interest/attachments/20200819/0979f920/attachment.html>