[Interest] CBOR Questions
jhihn at gmx.com
Wed Mar 6 16:41:28 CET 2019
Perhaps you are correct, but this is an issue with the deserialization, and since these objects exist to be transferred (via CoAP, or other means) failing to include features that support the typical use cases is a failure of the format. We don't have infinite reads on disk because they are block devices, infinite socket reads because there is MTU, and it all gets buffered. Knowing when an object is complete and parsable. From the spec:
"The code for an encoder or decoder must be able to be compact in order to support systems with very limited memory, processor power, and instruction sets."
So the approved way we have to repeatedly attempt to deserialize on "systems with very limited [...] processor power"?
So I looked at the CoAP protocol, and it says:
"The payload data extends from after the marker to the end of the UDP
datagram, i.e., the Payload Length is calculated from the datagram size."
" CoAP message, appropriately encapsulated, SHOULD fit within a
single IP packet (i.e., avoid IP fragmentation) and (by fitting into
one UDP payload) obviously needs to fit within a single IP datagram.
If the Path MTU is not known for a destination, an IP MTU of 1280
bytes SHOULD be assumed; if nothing is known about the size of the
headers, good upper bounds are 1152 bytes for the message size and
1024 bytes for the payload size."
So the protocol it expects to work with assumes a message-based upper bound, and of trivial size. This is not good. In light of this, I would suggest a QCborDataStream class that implements efficient parsing.
> Sent: Wednesday, March 06, 2019 at 10:12 AM
> From: "Jérôme Godbout" <godboutj at amotus.ca>
> To: "Jason H" <jhihn at gmx.com>, "Thiago Macieira" <thiago.macieira at intel.com>
> Cc: "interest at qt-project.org" <interest at qt-project.org>
> Subject: RE: [Interest] CBOR Questions
> I think the CBOR is the (de)serialization, the length and CRC or any other features like that is part of your transport protocol and not the CBOR itself. You need to transfer the whole data first and then pass it to CBOR deserialization when all data is received, you could also split with start delimiter. I think you want to mix 2 different layers.
> -----Original Message-----
> From: Interest <interest-bounces at qt-project.org> On Behalf Of Jason H
> Sent: March 6, 2019 9:50 AM
> To: Thiago Macieira <thiago.macieira at intel.com>
> Cc: interest at qt-project.org
> Subject: Re: [Interest] CBOR Questions
> > Sent: Tuesday, March 05, 2019 at 11:42 PM
> > From: "Thiago Macieira" <thiago.macieira at intel.com>
> > To: interest at qt-project.org
> > Subject: Re: [Interest] CBOR Questions
> > On Tuesday, 5 March 2019 11:09:52 PST Jason H wrote:
> > > It seems the simplest, most reliable way to implement complete reads
> > > is to serialize the QByteArray data in a QDataStream. Unfortunately
> > > it looks like CBOR types do not have QDataStream support? If I were
> > > to request anything in terms of new API that would probably be it.
> > The simplest is to use the QCborValue::fromCbor overload that takes a
> > QByteArray (your buffer) and QCborParserError. Every time you receive
> > data, you try to parse and then you check the error
> > - if it's QCborError::EndOfFile, wait for more data
> > - if it's QCborError::NoError, remove "offset" bytes from the beginning of
> > the buffer and process the loaded map
> > - if it's anything else, give up
> That seems extremely CPU wasteful. While a QDataStream would know exactly when all the bytes are there and would parse the object in one go.
> But short of that, it seems maybe like we need a API that has update() like a Hashing algorithm, and a way to return a list of top-level objects with a function that much implements the loop with the logic you expressed.
> I'm not a fan of keep "trying until it works". Such protocols are never good. HTTP has content-length, or chunked encoding, but there's no "keep trying until it works" header. I realize this is more an issue with the CBOR standard not having a top-level item sentinel, and not a Qt implementation thing.
> Interest mailing list
> Interest at qt-project.org
More information about the Interest