[Interest] How to properly show progress for a signal processing pipeline

[Murphy, Sean]

We have a data processing pipeline that looks like so:
    Raw Data objects -> Type "A" data objects -> Type "B" data objects -> Final Data objects

Each step can be considered a filtering process where the following mathematical relationship holds for the quantities of each type of object:
    Raw count >= A count >= B count >= Final count
Technically all of those quantity relationships can be >=, but in practice they almost never equal, the left hand side is almost always greater than right hand side, but it's technically possible that some could be equal with the right input data.

 So the issue I'm having here is that I want to show the user some sort of progress indicator (this whole pipeline takes a bit of time), but at the start I only know the raw count, not any of the intermediate or final counts. And because the A->B->Final portion of the pipeline takes a noticeable amount of time, I can't just run the progress bar from 0 to Raw count, because then from the user's standpoint the progress bar would reach 100% once the number of raw samples is exhausted, not when processing is fully complete.

Is my only strategy to use the "undetermined state" of QProgress bar to just show "something" is happening, but not how far along we are in the process? To make the issue even a little more complicated, each processing node is running in a separate thread, so these conversions are happening in parallel as much as possible, making tracking a little more difficult. For example, at any given point in time, the node that converts type A objects to B objects, might be empty. It may have processed all of the A objects it has received into all the B objects it can make and passed them on to the next node, while the node ahead of it might not have finished its work on the next chunk of raw data to emit the next A object. So at that instant in time, the A->B node thinks it's "done" until it receives the next A object at some time later.

Sean


This message has been scanned for malware by Forcepoint. www.forcepoint.com