[Development] Linux event dispatcher work

[Robin Burchell]

Hi all,

As I'm starting to get somewhere with some of my recent hackery, I
thought I'd start discussing it a little more publically.

tl;dr if all goes well and I manage to actually clean up all this code
and get it reviewed, in the future, if you're on linux, hopefully the
core event processing will be a lot snappier. Any input is welcome, I
hope to start providing patches over this weekend.

I've been working on a new event dispatcher[1]
(QAbstractEventDispatcher subclass) in Qt's core. Specifically, I've
been working on an event dispatcher to best make use of Linux-specific
features (like eventfd, epoll, etc) and experimenting with that, with
a fair degree of success: I've got a prototype that can run pretty
quickly, faster than both QEventDispatcherUnix and
QEventDispatcherGlib by quite a fair margin[2] - as it's at quite an
early stage, I don't want to shout benchmarks from the top of the roof
at this point, but my superficial eventbench testcase, which fires off
QMetaObject::invokeMethod as fast as it can, is running at somewhere
around 20-30% higher throughput than either of those.

Anyway - now I've got this, I need to do something useful with it,
right? I happened to have an accidental pseudo-meeting with Thiago
about this just now, and as we're a cool open source project, that's
the whole reason I'm writing this mail. The general conclusion is that
we can try do it, but we do it in chunks, so here's my planned
roadmap:

- Split timer handling out of QUnixEventDispatcher for my saniy (done:
http://codereview.qt-project.org/#change,10683)
- Split QUnixEventDispatcher into two classes - a base containing
things like timer handling and other parts that Linux can share with
other unices, and an actual implementation containing the parts that
can't be shared - especially intra-thread wakeup (more on that in a
moment)
- Introduce a QLinuxEventDispatcher class (mostly a copy of the Unix
implementation class initially)
- Introduce eventfd(2) handling to QLinuxEventDispatcher (this is
much, much, much, orders of magnitude faster than the generic current
thread pipe solution)
- Introduce epoll (possibly optionally, possibly not, if I manage to
browbeat Thiago enough ;), in some yet-to-be-decided form agreeable to
all)
- timerfd seems to be a useful solution to avoid doing timers in
userspace/polling; I'm not sure on the practicality, needs more
investigation. (Thiago, I forgot to mention this)

In slightly more detail on some of points:

    eventfd
    =======

    With my initial unix-based dispatcher, with just epoll
implemented, I was somewhat surprised at the rather low call rates I
was getting, so I did a bit of digging, and found out that thread
waking was taking quite a long time. Since I was bored, I hacked
eventfd support in (it wasn't all that hard), and suddenly, the number
of QMetaObject::invoke()s I could do per second increased by a very
large number - around 100,000.

    epoll
    =====

    Epoll will allow Qt applications to scale to (much) larger numbers
of FDs than the current unix event dispatcher - which is stuck at
whatever FD_SETSIZE defaults to nowdays on Linux, something I don't
personally know off the top of my head anymore.

    I highly hope (and partially suspect) it will _also_ reduce
latency/etc (yes, even with the overhead of syscalls to disable
writability notifications) for smaller numbers of sockets, but this is
the part I need to prove somehow before I can make a strong case for
making epoll the default. Thoughts on this welcome.

[1]: For the uninitiated, an event dispatcher is the heart of a lot of
things. It receives events from various sources (like sockets, timers,
etc), and handles passing them off to Q(Core)Application, or wherever
else they need to go.
[2]:  (usual disclaimer: code is work in progress/probably buggy/incomplete)

Questions, comments, rocket propelled grenades?

BR,

--
Robin Burchell
http://rburchell.com