[Interest] json file handling

Jason H jhihn at gmx.com
Wed Apr 15 20:27:00 CEST 2015



> Sent: Wednesday, April 15, 2015 at 1:28 PM
> From: "Andre Somers" <andre at familiesomers.nl>
> To: interest at qt-project.org
> Subject: Re: [Interest] json file handling
>
> On 15-4-2015 17:25, Thiago Macieira wrote:
> > On Wednesday 15 April 2015 16:36:15 André Somers wrote:
> >>> The dilemma I'm in is, how can I tell the json writer to "append" to a
> >>> current json file without loading the whole file back in memory (eg
> >>> read json from file->edit in Qt->write back to file). Because, after a
> >>> while, I suppose the file will get quite large.
> >> I don't think JSON is particulary suitable for this kind of application
> >> (and neither is XML).
> > CBOR might be, since it does support "undefined length" maps and arrays.
> I'm not familiar with that format, but I'll look it up. I guess the 
> choice for JSON or XML is made due to convenience though, as classes to 
> read and write these are readily available. I doubt the same can be said 
> of CBOR.
> > As for XML, the format may not be suitable but QXmlStreamWriter is since it
> > keeps the state of which tags it needs to close and allows you to simply
> > stream data into it. On the receiver side, QXmlStreamReader is capable of
> > receiving incomplete data and making the best it can with what has been
> > completed.
> I'd expect a log-file like output to be well defined at any moment in 
> time, or as close as possible to that. So, while QXmlStreamWriter is 
> suitable, it does not produce a valid XML file until you finalize the 
> write. That may not be what you want (even though QXmlStreamReader could 
> be used to parse the incomplete file*).
> 
> I think a simpler aproach may be a better one in such cases: simply add 
> new values to the end of a file when the become available. No fancy 
> markup languages to wrap them. That will result in a smaller file 
> (especially compared to XML), faster writing and no problems reading the 
> data back in. Simple CSV will do if you want a text-based file, but a 
> binary file could also function really well if that is not needed. Just 
> append to the file when you have new data available. Such approaches 
> have worked with log files for decades :-)
> 
> André
> 
> *) Though I seem to remember there are actually some problems with that 
> in reality.


Back in the day I developed a generic serializer around the operations of push, item(key) value(value), and pop. I think it was called RecursiveMap and I think Thiago may have given it to me. It was based off QVariantMap, except there was a value() member. 

Anyway, when converting structured data, you can imagine everything as those four functions. 
push():  add an item (or key) to a list 
pop(): end the list 
item(key): create an item with key
value(value): set the value of the last item.

{"this": "is", "some": {"1": "JSON"} } = 
push().item('this').value('is').item('some').push().item('1').value('JSON').pop().pop()

You can then write trivial translaters from any format to any other format. XmlStreamReader/Writer is handy for XML. JSON is trivial as well, either from a object perspective or text stream perspective.

Textually:
push() = '{'
item(key) = '"key":'
value(value) '"value",' (or 'value,'), if value is int, or real, etc.
pop() = '}'

Objectually:
push() = obj={} // set currentObj  to new obj, [].push(obj) onto objectStack
item(key) = // set currentObj[key] to valueRef, currentObj[key]=null
value(value) // valueRef=value
pop() // pop from objectStack


Don't forget though with JSON, there is only ever one root element, which is a dictionary. You can't have your root JSON be and array [], or a bunch of items "1,2,3" or {},{},{}, they all must be in a {"root: _items_{}_or_[]__ } 
Which means you will always need a closing '}'




More information about the Interest mailing list