[Interest] Parsing data from serialport

[Roland Hughes]

On Monday April 8 22:59:23 CEST 2019 Martin Marmsoler wrote:

 >What is, when I ignore the ready read? Has the internal buffer a max 
size?
 >Because at the moment I have a memory leak and maybe it comes from that?

Martin,

Sorry to take so long in responding but this was going to be a long post
and I might have to bust it up in to multiple messages. I don't remember
what the maximum digest message size is. Easter Sunday is the only day I've
had enough time to write something like this. I'm also sorry you got so
much bad advice.
There is no way anyone who is actually a Dev would have missed

 >each line consists of multiple columns of data

So the sloppy non-solutions you got are a sad thing.

I'm going to start from the ground up so you understand the how and why of
things without any code getting in the way. I'm also going to do this in a
basic text editor (KWrite) because HTML gets stripped out of digest email
and the archive so any email with HTML formatting which actually had a good
answer is lost to the sands of time.

There are two primary architectures, some call them schools of design and
others call them design patterns when it comes to serial communications.

Transient Comm - It doesn't really matter because you only use a tiny
piece of the data. No, the "solution" you were provided cannot be
justified even for this.

Persistent Comm - Every packet matters. Full hardware flow control is
mandatory. Software flow control is forbidden. Senders are designed to
stop operation when one or more pins drop. (If the collector/controller
crashes they don't continue operation.)

You can find the current DB-9 pin definitions here:
http://www.usconverters.com/index.php?main_page=page&id=61

It's important to note the DB-25  pinout came from the industrial
revolution, long before we had computers.

Transient Comm is most often found in systems using truck scales. All of
the packets are discarded until an operator hits a key or clicks on
something which either starts a new ticket transaction or looks to
complete an existing ticket transaction by adding a tare weight.

Settling Read/Weight

You don't get just one weight from a truck scale. It will bounce around
for a long time. Fuel sloshing in tanks, breeze catching side of trailer,
driving rummaging around in cab for phone charger, the list goes on and
on. Windy days mean the scale will never get to a single value.

What your application has to do is have all of the settling rules
configured in an indexed file or database table, NOT in INI, JSON, XML or
some other text file. (The reason why comes much later.)

Every scale at every company (unless they all happen to be the exact same
model and brand) will have different settling rules. Even if they are all
the same model and brand they will have different settling rules when
weighing different products.

https://www.beverlymaterials.com/sand-and-gravel-prices/
ftp://www.ilga.gov/jcar/admincode/008/00800600ZZ9998bR.html

Pea gravel at $11/ton really doesn't matter if you allow the spread
between high and low weights over time be 500 pounds. As you can see bi
the ilga.gov link, where a bushel of spinach is 12 pounds, the settling
weight range has to be very tight.

Settling is ordinarily range over time. In the computer world the time is
usually a second or less. You must keep track of the highest read and the
lowest read for the length of settling time. If the difference is less
than the allowed spread your settled reading is exactly in the middle.

Some places and products are very very strict. Settling is N-reads in a
row which are "the same." Time is unlimited. This form of settling is
event driven. "The Same" can be exactly identical or it can be plus or
minus a small percentage from the initial read.

In a Transient Comm world most reads don't matter. The serial reading
still has to be done in its own thread. Packet processing also has to be
done in its own thread. What really differs between Transient and
Persistent is the hand off from the reading thread to the rest of the
application.

In a Persistent Comm application you aren't allowed to lose a single read.
Hardware flow control is required so the sender will shut down when the
receiver goes dead. Transient Comm basically throws the read away at this
point unless you are in that time window of a transaction. Persistent Comm
has to record it, usually to a local fake database on disk or battery
backed RAM. There is a timestamp, processed flag set to false and usually
the port number/ID as the primary key with an alternate key of processed
flag and timestamp.

Once successfully written a signal of some kind (Qt signal, DBUS,
whatever) to the process control thread which could be the main event loop
in a Qt application. Worker threads can have a long run-time depending on
how many data sources are required to process a packet. The process
control thread is tasked with spinning up however many worker threads have
been configured based on the depth of the unprocessed queue. (Count of
records with processed flag set to false.) Each worker thread will shut
itself down when it reaches into the fake database for more work and finds
none available.

That processed flag is not a boolean. It has at least 3 states if your
fake database doesn't implement record locking.

false
processing
true

The dynamic creation and deletion of worker threads is why you cannot use
text file based configuration storage. One of the first things a worker
thread has to do when processing a packet is retrieve the packet
configuration information from that config file/table. Bounding
characters, CRC algorithm (if any), is it a fixed length packet? Is there
both a checksum and CRC? Special character restrictions? etc. The last
thing you want is someone having that entire text file loaded for change
and in the process of rewrite when a worker thread starts. Bad things
happen. Fake databases/indexed files which implement record level locking
mitigate this risk because the worker thread can sleep waiting for record
lock release.

No matter what anyone tells you, always, always, always design your serial
collector to support multiple serial ports. They don't mean to, but they
lie. It may take a day, it may take a year, but once the first one is
running they will want to add something else. You don't want to start over
or tell them you have to start over.

Did a touch screen system using a raspberry pi to communicate with an
existing single board computer in a retail distribution machine. We hooked
onto the pins for the serial port. They swore up and down none of those
machines ever had more than one. After I left the owner saw how everyone
liked the touch screen and decided they should also put that solution in
their inventory management machines. These machines are for labs where an
employee can scan their badge and select some kind of test equipment or
tool. They can only have one. When they return the battery powered
equipment they have to physically hook up the charger before the door will
lock and give them credit. It is much more complex than a standard retail
machine. Most importantly, I think they said it has 4 of those other cards
in it. No idea how/if they solved that issue. I wrote the software to
handle it, despite being told not to, but the existing design already used
some of the USB ports.

Anyway, these are the two standard designs for serial collectors, no
matter what name you give them. I responded in raw text form here so it
would make it into the archive and be searchable.


on Fri Apr 5 21:01:52 CEST 2019 Konstantin Shegunov wrote:

 >Been there, done that. Be a lamb and stop with the nonsense. I'm sure you
 >run every file operation in a dedicated thread, 'cause you'd *"never ever
 >ever do I/O in the main GUI thread of a Qt application" ...*
 >Writing like a politician, with the all the generalizations without any
 >context, and context is everything, ain't impressing nobody here. You're
 >talking to devs, not to some old lady in the mall.

Sadly, the little old lady at the mall would have more dev skills than
you've demonstrated here.

You ain't impressing nobody here.


-- 
Roland Hughes, President
Logikal Solutions
(630)-205-1593  (cell)
http://www.theminimumyouneedtoknow.com
http://www.infiniteexposure.net
http://www.johnsmith-book.com