[Development] need to handle touch events rather than depending on mouse event synthesis

[Shawn Rutledge]

We've been chatting some more around the Oslo office about the fact that
even on platforms where the touchscreen is the primary pointing device,
both QWidgets and Qt Quick components are handling mostly mouse
events. In order to make that work, the Qt 4 approach (which is still
in place now) is for each incoming touch event, first let it propagate
as a touch event; then, if the event was not accepted, synthesize a
mouse event and let that propagate through. This method of achieving
backwards compatibility discourages handling touch events though: if
any variety of button widget or QML component, which handles mouse
events only, is a child (or descendant) of any component which handles
touch events, then when the touch event is accepted by the parent
component, the mouse event will not be synthesized. So the button (or
other mouse-only component) cannot be pressed.

The WebKit team has this problem, in that they want the QML web view to
be flickable, but obviously the user needs to be able to interact with
any mouse-only components (such as buttons) which might be on top. So
WebKit needs to separately synthesize mouse events from touch events
because the main synthesis mechanism doesn't work in that case.

In src/quick/items/qquickcanvas.cpp, there is a very recent new method
translateTouchToMouse which generates a new QQuickMouseEventEx event
type containing velocity information. (In fact maybe it makes sense to
just put the velocity in the base QMouseEvent, but that's optional to
what I'm about to propose.) This is IMO another case where mouse event
synthesis should not be necessary.  I suspect the reason for it is the
same as the WebKit case.

Graphics View has yet another way, but handling touch events there is
lower-priority than for QML.

If we set aside all the Qt history and think about what would have been
ideal if we were starting over from scratch, I think I'd have wanted a
"pointing" event type which has a bit less than what QMouseEvent does:
at least the coordinates and an enum for "what happened" (pressed,
released, clicked, double-clicked, entering, leaving and so on). The
mouse event could inherit that and add mouse-specific concepts like
having multiple buttons, and the touch event could inherit and add the
multiple-finger concept. The point being that naive widgets like
Button should not need to care where the event came from, just that it
was clicked or tapped, but not dragged; so QPushButton would just handle
the hypothetical Pointing event. Then most of the third-party legacy
apps would have already been doing the same thing, and we wouldn't have
trouble at this stage to introduce a touch event as a different
specialization of the Pointing event. (Alternatively multiple fingers
could be treated just like multiple mice: separate press/release events
for the independent fingers. But actually it's useful to group multiple
touch points together as long as they come from the same user; it
facilitates gestural UIs, in that the UI does not need to gather up
multiple points from multiple events occurring at different times, and
figure out that they are part of one gesture.)

Anyway, back to reality... my next idea was let's have a flag to enable
the mouse event synthesis, which should be true by default, so that we
can at least turn it off and try to develop pure-touch UIs. But it turns
out this flag already exists: AA_SynthesizeMouseForUnhandledTouchEvents,
which is true by default. And there is even the reverse one:
AA_SynthesizeTouchForUnhandledMouseEvents. So that's a good start.

The proposal is this: I think we need to have a QML PointingArea element
which looks just like MouseArea except that it handles both mouse events
and single-touch events the same way.  Then we need to start using it
instead of MouseArea in almost every case.  That way
AA_SynthesizeMouseForUnhandledTouchEvents can eventually be set to false
for some applications. 

We also need the exsisting QWidgets to start handling touch events too.
After that is done, individual QWidget-based apps in the field
(especially those with custom widgets) can set the
AA_SynthesizeMouseForUnhandledTouchEvents flag or not, depending on what
works better for them; but we need to move towards a future in which we
do not need to synthesize mouse events.

Some apps may eventually have a use for the distinction between mouse
and touch, too. One reason I got interested again at this particular
time is that I was thinking it would be nice if the KDE Konsole
(terminal application) was flickable. But dragging with the left mouse
button is the way that you select text, and that is also useful. So on
my touchscreen laptop, I can select text by dragging, regardless whether
I drag with a finger on the screen, with the touchpad, or with an
external mouse; but with a finger on the screen, selecting text is not
really what I expect. If Konsole handled both mouse events and touch
events, it could do something appropriate for each of them, and there
could be some useful multi-finger gestures too. This is just an example,
and in fact it may already be possible to implement this use case
without changes to Qt (except for the lack of support for XInput 2.2
and/or direct support for the evdev driver on my laptop, which I'm also
interested in looking at separately.)  But in a more complex case which
has more widgets or Qt Quick elements, if touch events are being
accepted at all, you need to have them understood everywhere.

But we have the issue that the QML MouseArea component is in very
widespread use. This is because QML started life on the desktop. There
is already QQuickMultiPointTouchArea and QQuickPinchArea; so in
applications which intend to be portable between touchscreen devices and
conventional desktop usage, it should be OK to have overlapping touch
areas and MouseAreas, and this will enable the app developer to
customize the behavior depending on whether the user is interacting with
a mouse or a finger. But naive components like any of the many Button
implementations should not necessarily need to care.  They should be
using the proposed PointingArea instead of MouseArea.

Alternatively MouseArea could handle touch events itself, but then
pretty soon we will start thinking the name sounds rather dated.  It
wouldn't even surprise me if people stop using mice in a decade or so;
whereas we will probably always have some kind of "pointing device",
thus the need for a generic name which won't be obsolete later.  And, we
still need a mouse-only Area which can be used in combination with the
touch-only Areas so that it's possible to make a cross-platform UI.

In summary the consequences of mouse event synthesis present some real
problems, and I think we need to get the device-agnostic PointingArea
into Qt5 ASAP.

-- 
MVH, Shawn Rutledge ❖ "ecloud" on IRC