[Development] Multiclient UDP server

[Thiago Macieira]

Related to the DTLS discussion, but I'll try to keep this free from crypto for 
now.

The question is how to write a multi-client server operating on UDP, not TCP. 
It would be nice to understand SCTP case too (and maybe DCCP in the future).

On TCP, this is easy:
1) you create a socket, bind it and then put it in LISTENING mode
2) every time a new TCP connection arrives, you accept() a new socket
3) the server socket is from now on independent of the client

In our API, the QTcpServer creates new QTcpSockets and they're from that point 
on independent.

In UDP, that's different. There are two ways to proceed:

A) single UDP socket in unconnected mode

The code identifies each different client by the source address. 
QNetworkDatagram makes this easy, since it holds for you the source address 
and using makeReply() creates a QNetworkDatagram that you can use to reply.

Pros: uses a single socket, so limits the resource usage.

Cons: scalability. Since it's the same socket, only one thread can read from 
it at a time. You may pass the QNetworkDatagram to another thread to process 
the data, but you need to centralise sending of replies (QUdpSocket 
limitation, not of the OS API).

We don't need to implement anything to support this: that is simply QUdpSocket 
as it is today.

B) multiple UDP sockets

This is similar to TCP. Whenever a new client is detected, we create a new UDP 
socket and put it in connected mode with that particular client. Once you do 
that, this socket will only receive datagrams from that client, whereas the 
unconnected socket will not.

Pros: scalability, since you can move the QUdpSocket to another thread; the 
connected datagram socket has another advantage that it can relay ICMP errors 
like "port unreachable" to the upper level, whereas the unconnected one won't. 
I also believe the connected mode socket can do Path MTU calculation.

Cons: more resources, but more importantly, there's a race condition: the 
unconnected socket which you received the initial packet from the new client 
on may have queued more datagrams. Those datagrams can be:
 - from the same client
 - from other clients

So we are in a catch-22 situation: if you connect() this socket and create a 
new unconnected one, any packets from other clients will not be seen. If you 
create a new socket and connect() it, you may miss any packets from the same 
client. I think that for both CoAP and DTLS, we'd opt for connect()ing the new 
socket, since the client wouldn't be sending new commands until it receives a 
reply from us. That reply resolves the problem of race condition.

[food for thought: what happens if we receive a retransmission?]


So questions:
 - do we attempt to provide a "QUdpServer" or augment QUdpSocket's API to 
support (B) above? Or do we provide only a QDtlsServer API that implements 
(B)? This "QUdpServer" would be useful to implement a unencrypted CoAP server 
("coap"). It's not very difficult to rewrite the code, but the choice becomes 
hairy if we want to support both unencrypted and encrypted CoAP ("coaps").

 - do we need a QDtlsSocket at all, or just a wrapper on top of QUdpSocket? 
One advantage of having the class is that you could implement both "coap" and 
"coaps" clients on top of a plain QUdpSocket pointer, not having to worry  
about the encryption.

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center