[Development] need to handle touch events rather than depending on mouse event synthesis

[Luciano Wolf]

Hi Frederik,

How the patches are going? I've interest on this touch related things
as we're having problems with Snowshoe webbrowser running on a N9. I'm
trying your patch (the 50th) against latest valid Qt5 hash but it's
not working. Should I try with any specific hash?

Cheers,
Luciano



On Tue, May 15, 2012 at 1:12 PM, Frederik Gladhorn
<frederik.gladhorn at nokia.com> wrote:
> Hi,
>
> it's been a while and the touch handling issues keep comming back.
>
> Since I don't think just talking helps I spent the last month writing patches
> with Shawn's and Laszlo's help, testing on actual touch screens.
>
> The result are two patches, I'd like to have more opinions on these. They are
> not 100% polished, but work quite well for me.
>
> The patches are bigger than I hoped, to sum them up, they "just change the
> event propagation of touch and sythesized mouse events to be parallel".
> So each item gets first offered the touch and if it claims to handle mouse
> events, also the mouse event.
> The tricky part is that items can filter their children's events to steal them
> (this is how a flickable works with a mouse area on top of it).
>
>
> Send mouse and touch in parallel:
> https://codereview.qt-project.org/#change,24189
>
> Make it work again with more than one touch point:
> https://codereview.qt-project.org/#change,25578
>
> (I don't mind squashing them, but it was easier to have one known good
> checkpoint while working on this)
>
>
> There are some issues still, for example currently you can only interact with
> one pinch area at a time.
>
> Basically my goal would be now to get the big behavior change in as soon as
> possible. In my opinion this is a now or never since we are way too far in the
> Qt 5 cycle already. I personally do think this is important and we'll always
> regret not getting this straight.
>
> Cheers
> Frederik
>
>
>
>
> Onsdag 29. februar 2012 17.20.36 skrev ext Shawn Rutledge:
>> We've been chatting some more around the Oslo office about the fact that
>> even on platforms where the touchscreen is the primary pointing device,
>> both QWidgets and Qt Quick components are handling mostly mouse
>> events. In order to make that work, the Qt 4 approach (which is still
>> in place now) is for each incoming touch event, first let it propagate
>> as a touch event; then, if the event was not accepted, synthesize a
>> mouse event and let that propagate through. This method of achieving
>> backwards compatibility discourages handling touch events though: if
>> any variety of button widget or QML component, which handles mouse
>> events only, is a child (or descendant) of any component which handles
>> touch events, then when the touch event is accepted by the parent
>> component, the mouse event will not be synthesized. So the button (or
>> other mouse-only component) cannot be pressed.
>>
>> The WebKit team has this problem, in that they want the QML web view to
>> be flickable, but obviously the user needs to be able to interact with
>> any mouse-only components (such as buttons) which might be on top. So
>> WebKit needs to separately synthesize mouse events from touch events
>> because the main synthesis mechanism doesn't work in that case.
>>
>> In src/quick/items/qquickcanvas.cpp, there is a very recent new method
>> translateTouchToMouse which generates a new QQuickMouseEventEx event
>> type containing velocity information. (In fact maybe it makes sense to
>> just put the velocity in the base QMouseEvent, but that's optional to
>> what I'm about to propose.) This is IMO another case where mouse event
>> synthesis should not be necessary.  I suspect the reason for it is the
>> same as the WebKit case.
>>
>> Graphics View has yet another way, but handling touch events there is
>> lower-priority than for QML.
>>
>> If we set aside all the Qt history and think about what would have been
>> ideal if we were starting over from scratch, I think I'd have wanted a
>> "pointing" event type which has a bit less than what QMouseEvent does:
>> at least the coordinates and an enum for "what happened" (pressed,
>> released, clicked, double-clicked, entering, leaving and so on). The
>> mouse event could inherit that and add mouse-specific concepts like
>> having multiple buttons, and the touch event could inherit and add the
>> multiple-finger concept. The point being that naive widgets like
>> Button should not need to care where the event came from, just that it
>> was clicked or tapped, but not dragged; so QPushButton would just handle
>> the hypothetical Pointing event. Then most of the third-party legacy
>> apps would have already been doing the same thing, and we wouldn't have
>> trouble at this stage to introduce a touch event as a different
>> specialization of the Pointing event. (Alternatively multiple fingers
>> could be treated just like multiple mice: separate press/release events
>> for the independent fingers. But actually it's useful to group multiple
>> touch points together as long as they come from the same user; it
>> facilitates gestural UIs, in that the UI does not need to gather up
>> multiple points from multiple events occurring at different times, and
>> figure out that they are part of one gesture.)
>>
>> Anyway, back to reality... my next idea was let's have a flag to enable
>> the mouse event synthesis, which should be true by default, so that we
>> can at least turn it off and try to develop pure-touch UIs. But it turns
>> out this flag already exists: AA_SynthesizeMouseForUnhandledTouchEvents,
>> which is true by default. And there is even the reverse one:
>> AA_SynthesizeTouchForUnhandledMouseEvents. So that's a good start.
>>
>> The proposal is this: I think we need to have a QML PointingArea element
>> which looks just like MouseArea except that it handles both mouse events
>> and single-touch events the same way.  Then we need to start using it
>> instead of MouseArea in almost every case.  That way
>> AA_SynthesizeMouseForUnhandledTouchEvents can eventually be set to false
>> for some applications.
>>
>> We also need the exsisting QWidgets to start handling touch events too.
>> After that is done, individual QWidget-based apps in the field
>> (especially those with custom widgets) can set the
>> AA_SynthesizeMouseForUnhandledTouchEvents flag or not, depending on what
>> works better for them; but we need to move towards a future in which we
>> do not need to synthesize mouse events.
>>
>> Some apps may eventually have a use for the distinction between mouse
>> and touch, too. One reason I got interested again at this particular
>> time is that I was thinking it would be nice if the KDE Konsole
>> (terminal application) was flickable. But dragging with the left mouse
>> button is the way that you select text, and that is also useful. So on
>> my touchscreen laptop, I can select text by dragging, regardless whether
>> I drag with a finger on the screen, with the touchpad, or with an
>> external mouse; but with a finger on the screen, selecting text is not
>> really what I expect. If Konsole handled both mouse events and touch
>> events, it could do something appropriate for each of them, and there
>> could be some useful multi-finger gestures too. This is just an example,
>> and in fact it may already be possible to implement this use case
>> without changes to Qt (except for the lack of support for XInput 2.2
>> and/or direct support for the evdev driver on my laptop, which I'm also
>> interested in looking at separately.)  But in a more complex case which
>> has more widgets or Qt Quick elements, if touch events are being
>> accepted at all, you need to have them understood everywhere.
>>
>> But we have the issue that the QML MouseArea component is in very
>> widespread use. This is because QML started life on the desktop. There
>> is already QQuickMultiPointTouchArea and QQuickPinchArea; so in
>> applications which intend to be portable between touchscreen devices and
>> conventional desktop usage, it should be OK to have overlapping touch
>> areas and MouseAreas, and this will enable the app developer to
>> customize the behavior depending on whether the user is interacting with
>> a mouse or a finger. But naive components like any of the many Button
>> implementations should not necessarily need to care.  They should be
>> using the proposed PointingArea instead of MouseArea.
>>
>> Alternatively MouseArea could handle touch events itself, but then
>> pretty soon we will start thinking the name sounds rather dated.  It
>> wouldn't even surprise me if people stop using mice in a decade or so;
>> whereas we will probably always have some kind of "pointing device",
>> thus the need for a generic name which won't be obsolete later.  And, we
>> still need a mouse-only Area which can be used in combination with the
>> touch-only Areas so that it's possible to make a cross-platform UI.
>>
>> In summary the consequences of mouse event synthesis present some real
>> problems, and I think we need to get the device-agnostic PointingArea
>> into Qt5 ASAP.
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development