[Qt-interest] How to deal with pointers for a QDataStream?

[Oliver Heins]

Hi André,

thank you for your kind help again.

Andre Somers <andre at familiesomers.nl> writes:

> On 20-4-2010 22:24, Oliver Heins wrote:
>
> Remember, there is nothing magical about QDataStream. It is reading
> and writing some binairy data, but it does not know anything about
> it's format that you did not tell it explicitly. Any byte in the
> stream is taking to mean exactly what you tell QDataStream it
> means. Also, the reading code knows nothing about the writing
> code. That means that you need to read it back exactly the way you
> wrote it out. So, you need to be very careful to keep your reading and
> your writing code symetrical. Also, you need to know what you expect
> in the datafile, and how much of it. That means for instance, that for
> every list you write out, you need to write out how many elements are
> in that list, and then be sure that you actually read that many
> elements. That is exactly what Qt is doing. So, it requires
> precision. It awards you with fast write-out and readback, and small
> files.
>
>>> Is token.unique actually a quint32? You stream it out as one, but I
>>> can not see if is streamed in properly based on this code.
>>>      
>> No, it is an ordinary int, and it seems to cause problems.  (I thought
>> there would be an implicit cast from quint32 to int, but I seem to be
>> wrong again.)
>>
> As I said above: you need to read back exactly what you wrote out. If
> you write out a quint32, you need to read back the same data type. The
> cast you as for does happen, but int is not casted to quint32, but (I
> guess) to qint32. Note the missing 'u' there. Having said that: why
> isn't token.unique a quint32 in the first place? Are you using
> negative identifiers?

I made it one, so no casting is needed for it anymore.  However, the
problem still remains.

,----
| class Token
| {
| public:
|     enum Type { Space, Paragraph, Kern, Char, Format, Note, End };
|     enum SubType { Null = 0,
|                    Bold, Italic,
|                    Header1, Header2, Header3, Header4, Header5, Header6,
|                    Itemize, Enumerate, Description, Item,
|                    Note1, Note2, Note3, Note4, Note5 };
| 
|     Token() { this->unique = ++lastUnique;
|         this->chr = (QChar)0;
|         this->subType = Null;
|         this->partner = 0;
|         this->parent = 0; }
|     // [...] (other constructors and destructor)
| 
|     Type type;
|     SubType subType;
|     QChar chr;
|     quint32 unique;
|     Token *parent;
|     QLinkedList<Token *> children;
|     Token *partner;
|     static quint32 lastUnique;
| };
`----

> Also, I find that it is a good idea to use Qt's explicit data types
> when you work with binairy files. The size of an int is not constant
> between systems, for instance. What do you expect to happen when you
> read in an int on a 32 bits machine from a file that was created on a
> 64 bits machine? If you are lucky, they agree on the size of int. If
> you are unlucky, then you are reading back only half of what was
> written out, and the next thing you read ends up being trash.

That's why I decided to cast to quint32.  But you're right, I should
make it an quint32 in the class itself.

>> Unfortunately, I can't get it work, even if I do an
>> explicit cast.  Here's what class Token now looks like and how the
>> operators are defined:
>>
> On first inspection, I don't see any obvious mistakes. Best is to do a
> bit of trying out. Write out your file, and open it with a binairy
> editor so you can inspect the data inside. There is a very nice on in
> KDE that can handle Qt data types, if you need it. You can then
> inspect if the data actually looks like what you expect it to
> be. Perhaps start with writing out and reading back a single Token
> object.

You're talking about Okteta?  I installed it, but I can't figure out how
it can handle Qt data types.  However, from what I can tell, everything
looks fine to me.  This is the binary written out by

,----
|     QDataStream out(&file);
|     out.setVersion(QDataStream::Qt_4_6);
| 
|     out << quint32(MagicNumber);
|     QLinkedListIterator<Token *> i(parser->tokenlist);
|     quint32 max = 0;
|     quint32 count = 0;
|     while (i.hasNext()) {
|         max = qMax(max, i.next()->unique);
|         count++;
|     }
|     out << max << count;
|     i.toFront();
|     while (i.hasNext())
|         out << *(i.next());
`----

with

,----
| QDataStream &operator>>(QDataStream &in, Token &token)
| {
|     quint32 partner, parent;
|     qint16 type, subType;
| 
|     token.parent = 0;
|     token.partner = 0;
|     in >> type >> subType >> token.unique >> parent >> partner >> token.chr;
|     token.type = static_cast<Token::Type>(type);
|     token.subType = static_cast<Token::SubType>(subType);
| 
|     return in;
| }
`----


Magic
Number  max nr    count

8C2B0CA6 00000022 00000022


Type/    
Subtype  uniq id  parent   partner  chr
 
00030000 00000001 00000000 00000000 0048  Char: H
00030000 00000002 00000000 00000000 0061  Char: a
00030000 00000003 00000000 00000000 006C  Char: l
00030000 00000004 00000000 00000000 006C  Char: l
00030000 00000005 00000000 00000000 006F  Char: o
00030000 00000006 00000000 00000000 002C  Char: ,
00000000 00000007 00000000 00000000 0000  Space
00040001 00000008 00000000 00000016 0000  Format:Bold, Partner: 0x16
00030000 00000009 00000000 00000000 0057  Char: W
00030000 0000000A 00000000 00000000 0065  Char: e
00030000 0000000B 00000000 00000000 006C  Char: l
00030000 0000000C 00000000 00000000 0074  Char: t
00030000 0000000D 00000000 00000000 0021  Char: !
00000000 0000000E 00000000 00000000 0000  Space
00040002 0000000F 00000000 0000001C 0000  Format:Italic, Partner: 0x1c
00030000 00000010 00000000 00000000 0049  Char: I
00030000 00000011 00000000 00000000 0074  Char: t
00030000 00000012 00000000 00000000 0061  Char: a
00030000 00000013 00000000 00000000 006C  Char: l
00030000 00000014 00000000 00000000 0069  Char: i
00030000 00000015 00000000 00000000 0063  Char: c
00060000 00000016 00000000 00000008 0000  End, Partner: 0x08
00000000 00000017 00000000 00000000 0000  Space
00030000 00000018 00000000 00000000 0067  Char: g
00030000 00000019 00000000 00000000 0065  Char: e
00030000 0000001A 00000000 00000000 0068  Char: h
00030000 0000001B 00000000 00000000 0074  Char: t
00060000 0000001C 00000000 0000000F 0000  End, Partner: 0x0f
00000000 0000001D 00000000 00000000 0000  Space
00030000 0000001E 00000000 00000000 0061  Char: a
00030000 0000001F 00000000 00000000 0075  Char: u
00030000 00000020 00000000 00000000 0063  Char: c
00030000 00000021 00000000 00000000 0068  Char: h
00030000 00000022 00000000 00000000 0021  Char: !


The code I use to read is:

,----
|     QDataStream in(&file);
|     in.setVersion(QDataStream::Qt_4_6);
| 
|     quint32 magic, max, count;
|     in >> magic >> max >> count;
|     if (magic != MagicNumber) {
|         QApplication::restoreOverrideCursor();
|         QMessageBox::warning(this, tr("Critical Text Editor"),
|                              tr("File %1 is not recognized.")
|                              .arg(file.fileName()));
|         return false;
|     }
| 
|     for (quint32 i = 0; i < count; ++i) {
|         Token token;
|         in >> token;
|         tokenVec.append(token);
|     }
`----

with 

,----
| QDataStream &operator>>(QDataStream &in, Token &token)
| {
|     quint32 partner, parent;
|     qint16 type, subType;
| 
|     token.parent = 0;
|     token.partner = 0;
|     in >> type >> subType >> token.unique >> parent >> partner >> token.chr;
|     token.type = static_cast<Token::Type>(type);
|     token.subType = static_cast<Token::SubType>(subType);
| 
|     return in;
| }
`----

What happens, is that the first 12 elements are read properly to token,
but from the 13th on, I get trash.  Debugging tells me, that then
token.unique is set to 0 and token.chr gets a value of (ushort)8448.
>From that on, only garbage is read, and everything, including
token.type.  So it looks like the data comes out of sync.

Best regards,
 olli