[Qt-interest] Faster logfile viewer

[Andre Somers]

Hi,

Christoph Bartoschek wrote:
> Arnold Krille wrote:
>
>   
>> You are calling repaint() which executes an immediate repaint. You should
>> call update() which schedules the paintEvents in the event-loop.
>> Replacing the repaint() with update() reduces the runtime significantly
>> (in callgrind) and results naturally in much lower costs and only one
>> effective paintEvent.
>>
>> In practice this change will not be a problem because normally the log-
>> messages are coming distributed over time and there will be enough time to
>> do a paintEvent every now and then.
>>
>> Unless you really want a log-_file_-viewer but in that case it should be
>> far simplier to load the file as whole into the widget. Or at least in
>> large chunks, not line by line.
>>     
>
> Hi,
>
> unfortunately using update() does not work. I have to explain the
> application a little bit more:
>
> It's a VLSI-application that executes operations that take several hours of
> runtime during which the GUI is effectively blocked. There is only the
> TextEdit with the logfile where the operations write status information to.
> Normally this works fine but sometimes the operations want to write more
> than 100000 lines in a very short time.
>
> The resulting requirements are:
>
> - Because one does not know when the next line has to be shown one cannot
> merge several lines together and has to show each one separately.
> - Because each line can be the last one for a longer time the widget has to
> be updated after each line.
>
> Our old GUI uses Motif and also colored lines. The updates are also after
> each line but writing 10000 lines takes less than one second. 
>
>   
Interesting problem, and worth playing around with...

If your calculations are blocking the GUI, would it not be time to 
separate the calculations to run in a separate thread? This way, you can 
have your GUI stay responsive (and use the normal update mechanisms).  
You can communicate your updates through signals & slots. Sending an 
insane amount of signals this way saturates the event loop, but with 
100000 lines on my system this is tolarable, while with 10000 it is 
barely noticable. I have attached my version of your code.

Three main modifications:
* used QPainTextEdit instead of QTextEdit (although it does not matter 
*that* much, from 13,8 down to 9,5 seconds)
* used a threading approach (made it 5x faster in my case, down to 2,6 
seconds)
* used a buffer and a timing mechanism to update the log view (almost 
30x faster, down to 0,094 seconds)
Overall gain: almost 150x faster.

The display is fast, but like I said, still can get a bit jerky if the 
eventloop saturates with log messages. Fixing that can be done in two steps:
1) trigger a buffer flush when the buffer count reaches a set limit
2) trigger a repaint in the buffer flush

Both cost time, but especially #2 makes the display much smoother for 
large sets.
Testing with 10,000 (not large!) items, I get 0,26 s runtime (still more 
than 2x than without), and with 2 enabled to this becomes 0,28; slightly 
more. Using a larger set (500,000 items), I get 13,8s without smoothness 
fixes, 21,6 with only fix 1, and 26,6 with fix 1 and 2. The latter is 
really smooth, and probably worth the time spend. Overall speedup 
relative to the original version is still 50x faster.

Play around with it using the attached files. Enable the above fixes by 
commenting or uncommenting them in ama.hpp.

Timing notes: they are really unscientific, based on one or two runs 
only per combination. All runs on a Intel Core2 Quad 9550 @ 2,83GHz.

Note that you can use the update buffering tricks without threading too. 
I hope this inspires you to make some further modifications to speed it 
up even more.

André


-------------- next part --------------
A non-text attachment was scrubbed...
Name: logtest.zip
Type: application/x-zip-compressed
Size: 1509 bytes
Desc: not available
Url : http://lists.qt-project.org/pipermail/qt-interest-old/attachments/20090805/581461de/attachment.bin