[Qt-interest] QtScript signal/slot thread issues

[Russell Ryan]

Hi again,

Just writing with an update: We were able to fix the crash but by moving 
away from using qScriptConnect. Instead now we emulate qScriptConnect 
with a map of objects to functions in the script engine, and we connect 
the valueChanged signal to the MidiScriptEngine. Once we did this, we 
started seeing QEvent::MetaCall events in our event loop, and the slot 
itself was firing in the proper thread. Now we manually call the script 
functions from the slot in MidiScriptEngine.

We would really like to use qScriptConnect for performance reasons 
(before the switch we were seeing much better interactivity of our MIDI 
console and the script engine).  Is there a way that we can use 
qScriptConnect and have the 'slots' fire in our MidiScriptEngine 
QThread's event loop? All I can surmise now is that the slots are either 
being delivered directly, or they are being queued in some other event 
loop (e.g. the main application's).

Thanks in advance for your help,
RJ Ryan

Russell Ryan wrote:
> Hi Kent,
>
> I'm RJ Ryan, another Mixxx developer working with Sean on the QtScript 
> bugs we're having.
>
> First to summarize:
>
> - We've eliminated some bugs by following your suggestions to avoid 
> multiple threads using the QScriptEngine at once
> - We still have a crash bug that seems to be related to the 
> QScriptEngine. It takes more effort to produce, but it amounts to 
> generating a lot of MIDI signals (rapidly moving the sliders/knobs on 
> the device up and down).
>
> MidiScriptEngine, our class that wraps QScriptEngine, is now a 
> QThread, which in its run() method, creates the QScriptEngine, then 
> calls exec(), so that the event loop starts running. Our goal is that 
> all of our interactions with the QScriptEngine go through this thread.
>
> The former 'execute()' method on MidiScriptEngine (which previously 
> executed QScriptEngine::call() directly) now generates a custom 
> QEvent, which is captured by the MidiScriptEngine::event() method, 
> which branches to safeExecute, a private method which does the 
> QScriptEngine::call method. We have verified that the Thread ID of 
> MidiScriptEngine::event() calls is the same as that of the 
> MidiScriptEngine::run() method.
>
> When MidiScriptEngine is created via the main application thread, we 
> immediately call moveToThread() on itself, so that its thread context 
> is switched to its own thread. I was unsure if this was necessary.
>
> Now there are 3 main ways that something can happen to the engine:
> 1) a MIDI device produces a signal, which is received by a MidiObject 
> QThread, and results in an MidiScriptEngine::execute() call, which is 
> marshalled into the MidiScriptEngine thread via the machinery I 
> described above.
> 2) A ControlObject emits a valueChanged() signal. If the control 
> object's valueChanged() signal is qScriptConnect()'d to a script 
> function, then the QScriptEngine calls the function once the slot 
> arrives.
> 3) All scripts are provided a reference to the MidiScriptEngine 
> object, and they are able to call MidiScriptEngine::connectControl(), 
> which results in a qScriptConnect or Disconnect.
>
> In your previous emails you said that qScriptConnect uses 
> AutoConnections for the connections. In order to maintain our mutual 
> exclusion of engine use, we would like the slot side of the 
> qScriptConnect to execute in the MidiScriptEngine thread.  People in 
> #qt told me that QueuedConnections slots or AutoConnection slots 
> across threads would show up in an event() method as a 
> QEvent::MetaCall event. We have tested this, and it does not seem to 
> be the case. Signals that are emitted are arriving at the script 
> slots, but not showing up in the MidiScriptEngine::event() method. (We 
> simply print every event, handle our custom events, and return the 
> default implementation otherwise).
>
> Question 1
> =====
> We have verified that all ControlObjects are created in the main 
> application thread. We have also verified that the QScriptEngine is 
> created in the MidiScriptEngine thread.
> Is it true that under these circumstances, an AutoConnection should 
> queue all of its slots in our event loop, instead of being directly 
> fired? Is it possible to test this?
>
> Question 2
> =====
> Is it safe for a QtScript to invoke a function that would affect the 
> engine? In our case, a QtScript can make a call to the 
> MidiScriptEngine::connectControl, which results in a qScriptConnect on 
> the engine. Would this re-enter the QScriptEngine ? The callstack 
> would be something like QScriptEngine::call()  -> (script) -> 
> MidiScriptEngine::connectControl() -> qScriptConnect().  What if that 
> ended up call()'ing something instead of qScriptConnect()?
>
> Characteristics of our crash
> =====
> If you disable qScriptConnects (i.e. valueChanged signals won't reach 
> the script) then the crash goes away.
> If you disable execute() calls from the MIDI signals coming from the 
> device, then the crash goes away.
>
> Some other notes
> =====
> - execute() calls can never occur concurrently because only one other 
> thread delivers them
>
> All work on this issues is now going on in this branch:
> https://mixxx.svn.sourceforge.net/svnroot/mixxx/branches/Features_MIDIScriptAutoReaction/ 
>
> If you'd like to build it, we have a special MIDI xml/script 
> crashTest.xml/js, which should allow you to easily crash it if you 
> have a MIDI controller.
>
> Here are a number of backtraces. In some cases the MidiObject thread 
> is listed as crashed, while in others the MidiScriptEngine thread is 
> listed as crashed. When the MidiObject thread is crashed, it appears 
> as if the MidiScriptEngine thread has been corrupted.
>
> http://pastebin.com/d49184d0
> In this backtrace, the MidiScriptEngine thread has crashed.
>
> http://pastebin.com/d723da46c
> In this one, the MidiObjectAlsaSeq thread (LWP 3594) has crashed, and 
> the thread directly below it (the MidiScriptEngine thread LWP 3593) is 
> listed as corrupted.
>
> http://pastebin.com/d17db795f
> Here MidiObjectAlsaSeq has crashed, but the MidiScriptEngine thread 
> seems fine.
>
> We'd appreciate any insight you can provide, since at this point we're 
> stuck again.
>
> Thanks very much,
> RJ Ryan
>
>
> Sean M. Pappalardo wrote:
>> Kent's responses attached
>>
>> <<--------------------------------------------------------------------------------->> 
>>
>> This E-Mail message has been scanned for viruses
>> and cleared by >>SmartMail<< from Smarter Technology, Inc.
>> <<--------------------------------------------------------------------------------->> 
>>
>>  
>> ------------------------------------------------------------------------
>>
>> Subject:
>> Re: [Qt-interest] QtScript segfault on C++ signal (dis)connection/firing
>> From:
>> Kent Hansen <khansen at trolltech.com>
>> Date:
>> Wed, 14 Jan 2009 17:44:42 +0100
>> To:
>> qt-interest at trolltech.com
>>
>> To:
>> qt-interest at trolltech.com
>>
>>
>> Hi Sean,
>> Any chance that the same script engine is being accessed from multiple
>> threads? QtScript is not thread-safe, so be sure that there's only a
>> single thread evaluating code (unless you've explicitly added a
>> synchronization mechanism), and make sure that all connections are
>> queued. If QtScript intercepts a signal and starts to handle it, and
>> then another thread starts evaluating a script while this is going on,
>> one likely result is that the QtScript call stack will get corrupted,
>> which the backtrace seems to indicate.
>>
>> Regards,
>> Kent
>>
>>
>> Sean M. Pappalardo wrote:
>>  
>>> Hello again.
>>>
>>> Just wondering if you've been able to find anything out? This issue is
>>> quite literally a show-stopper, considering this is a DJ app. :)
>>>
>>> After a bunch more testing, the crashing doesn't happen if I don't
>>> connect any of the C++ signals. I can call MidiScriptEngine::setValue()
>>> all day from the script with no problems. It's crashing when signals 
>>> are
>>> connected and setValue() is called many times in a row, affecting
>>> objects that emit signals. And it's worse on faster systems (more
>>> setValue()s per second, I assume.) The signals only fire once per
>>> latency period though. I did also find that increasing the latency alot
>>> (400ms or more) makes it less likely to happen, depending on processor
>>> speed.
>>>
>>> I ran valgrind with Mixxx compiled against Qt 4.5 which seems to give
>>> more detail in the QtScript area, incase it helps:
>>>
>>> =20125== Process terminating with default action of signal 11 (SIGSEGV)
>>> ==20125==  Access not within mapped region at address 0x8
>>> ==20125==    at 0x530F078:
>>> QScriptContextPrivate::execute(QScript::Code*) 
>>> (qscriptcontext_p.cpp:2020)
>>> ==20125==    by 0x5320B15:
>>> QScript::ScriptFunction::execute(QScriptContextPrivate*)
>>> (qscriptcontext_p.cpp:313)
>>> ==20125==    by 0x5335B7A: QScriptEnginePrivate::call(QScriptValueImpl
>>> const&, QScriptValueImpl const&, QList<QScriptValueImpl> const&, bool)
>>> (qscriptengine_p.cpp:1252)
>>> ==20125==    by 0x5371496: QScriptValue::call(QScriptValue const&,
>>> QList<QScriptValue> const&) (qscriptvalueimpl_p.h:719)
>>> ==20125==    by 0x80FA4C1: MidiObject::receive(MidiCategory, char, 
>>> char,
>>> char, QString) (midiobject.cpp:273)
>>> ==20125==    by 0x81BA89F: MidiObjectALSASeq::run()
>>> (midiobjectalsaseq.cpp:286)
>>> ==20125==    by 0x502861D: QThreadPrivate::start(void*)
>>> (qthread_unix.cpp:184)
>>> ==20125==    by 0x42CB4BF: start_thread (in
>>> /lib/i686/cmov/libpthread-2.7.so)
>>> ==20125==    by 0x55B461D: clone (in /lib/i686/cmov/libc-2.7.so)
>>>
>>> Thanks again for your time. Let me know your thoughts and if there's
>>> anything else I can do to help.
>>>
>>> Sincerely,
>>> Sean M. Pappalardo
>>>
>>> <<--------------------------------------------------------------------------------->> 
>>>
>>> This E-Mail message has been scanned for viruses
>>> and cleared by >>SmartMail<< from Smarter Technology, Inc.
>>> <<--------------------------------------------------------------------------------->> 
>>>
>>>       
>>
>> _______________________________________________
>> Qt-interest mailing list
>> Qt-interest at trolltech.com
>> http://lists.trolltech.com/mailman/listinfo/qt-interest
>>
>> <<--------------------------------------------------------------------------------->> 
>>
>> This E-Mail message has been scanned for viruses
>> and cleared by >>SmartMail<< from Smarter Technology, Inc.
>> <<--------------------------------------------------------------------------------->> 
>>
>>  
>> ------------------------------------------------------------------------
>>
>> Subject:
>> Re: [Qt-interest] QtScript segfault on C++ signal (dis)connection/firing
>> From:
>> Kent Hansen <khansen at trolltech.com>
>> Date:
>> Thu, 15 Jan 2009 18:13:43 +0100
>> To:
>> qt-interest at trolltech.com
>>
>> To:
>> qt-interest at trolltech.com
>>
>>
>> Hi Sean,
>>
>> Sean M. Pappalardo wrote:
>>  
>>> Hello again.
>>>
>>> Kent Hansen wrote:
>>>      
>>>> Hi Sean,
>>>> Any chance that the same script engine is being accessed from multiple
>>>> threads?           
>>> That's actually almost certain. I'm pretty sure the emitting object is
>>> in a different thread than where the engine was instantiated.
>>>
>>>      
>>>> QtScript is not thread-safe, so be sure that there's only a
>>>> single thread evaluating code (unless you've explicitly added a
>>>> synchronization mechanism),
>>>>           
>>> A single thread ever or one thread at a time?
>>>       
>>
>> One thread at a time per QScriptEngine. This also applies to operations
>> on QScriptValues (e.g. QScriptValue::call()), since those values are
>> bound to a particular engine.
>>
>>  
>>>      
>>>> and make sure that all connections are
>>>> queued.
>>>>           
>>> Signal connections? Can you point me to info on how to do that? (I'm
>>> relatively new to C++.)
>>>
>>> Thank you again very much for your time! This information will help 
>>> alot!
>>>       
>>
>> Connections created with QtScript (e.g. qScriptConnect()) are
>> auto-connections; see
>> http://doc.trolltech.com/4.4/qt.html#ConnectionType-enum.
>> If multiple threads are emitting signals to be caught by a single script
>> engine, you should ensure that the objects emitting the signals don't
>> live in the script engine thread, and ensure that calls to
>> QScriptEngine::evaluate() (or any other operation that can have a
>> side-effect on the engine, e.g. QScriptValue::toString()) are done from
>> the script engine thread (or synchronized properly -- but I recommend
>> the queuing approach). See http://doc.trolltech.com/4.4/threads.html and
>> http://doc.trolltech.com/4.4/qobject.html#moveToThread.
>>
>> Regards,
>> Kent
>> _______________________________________________
>> Qt-interest mailing list
>> Qt-interest at trolltech.com
>> http://lists.trolltech.com/mailman/listinfo/qt-interest
>>
>> <<--------------------------------------------------------------------------------->> 
>>
>> This E-Mail message has been scanned for viruses
>> and cleared by >>SmartMail<< from Smarter Technology, Inc.
>> <<--------------------------------------------------------------------------------->> 
>>
>>  
>> ------------------------------------------------------------------------
>>
>> Subject:
>> Re: [Qt-interest] QtScript conceptual questions
>> From:
>> Kent Hansen <khansen at trolltech.com>
>> Date:
>> Thu, 15 Jan 2009 18:40:13 +0100
>> To:
>> qt-interest at trolltech.com
>>
>> To:
>> qt-interest at trolltech.com
>>
>>
>> Hi Sean,
>>
>> Sean M. Pappalardo wrote:
>>  
>>> Hello again.
>>>
>>> I have a few high-level questions about QtScript since it seems my
>>> understanding is seriously flawed:
>>>
>>> 1) Does one need to evaluate() an entire script file before calling
>>> individual functions contained within? (If not, would it hurt to do so?
>>> The point being to report syntax errors at the start of the program
>>> instead of at call time.)
>>>       
>>
>> evaluate() returns a SyntaxError object if there is a syntax error in
>> the script; you can get a string representation by calling toString() on
>> it, or access the properties of the object individually (e.g.
>> lineNumber, message).
>> If there _is_ a syntax error in the script, then no part of the script
>> is actually going to get evaluated (the only exception to this rule are
>> statements like "0 = 0" and "++0", which will throw a syntax error when
>> (if) that statement is actually reached during evaluation).
>>
>>  
>>> 2) When one does QScriptValue slot = execute(functionname); what's
>>> actually happening? Are we making a self-contained object that is the
>>> function? Or simply referencing its instance in the ScriptEngine?
>>>       
>>
>> What does execute(functionname) do?
>> If you mean something like
>>
>> QScriptValue fun = engine.evaluate("function foo() { return 123; }; 
>> foo");
>>
>> Then yes, the C++ variable "fun" will now hold a reference to the script
>> function "foo".
>> "fun.call()" in C++ will achieve the same as doing "foo()" in a script.
>>
>>  
>>> 3) When one then does qScriptConnect(object, SIGNAL(signalName()),
>>> QScriptValue(), slot); what's actually happening here? Are we 
>>> connecting
>>> the signal to a self-contained object?
>>>       
>>
>> There's a magic C++ object behind the scenes that connects to the
>> signal. When the signal is received, the C++ arguments are converted to
>> script values and the function you passed as the last argument to
>> qScriptConnect() is called with the converted arguments.
>>
>>  
>>> 4) What implications does all this have for multi-threaded code?
>>> Especially considering
>>> http://doc.trolltech.com/4.4/qtscript.html#controlling-qobject-ownership 
>>>
>>>       
>>
>> See my latest reply in the "QtScript segfault on C++ signal
>> (dis)connection/firing" thread. Object ownership doesn't matter here,
>> thread affinity does.
>>
>>  
>>> 5) What's the best way to protect against one thread evaluate()ing some
>>> function and a connected signal in another thread evaluate()ing 
>>> another?
>>>       
>>
>> Again, see my previous reply. Because of the way signal handling works,
>> you'd have to not only synchronize the explicit calls to QtScript
>> functions, but also your signal emissions -- which would be a rather
>> unworkable solution. Use queued connections, use
>> QMetaObject::invokeMethod(), have threads communicate through events;
>> see e.g.
>> http://doc.trolltech.com/4.4/eventsandfilters.html#sending-events. In
>> this way you rely on the (thread-safe!) primitives provided by Qt to
>> serialize access to the script engine.
>>
>> Regards,
>> Kent
>> _______________________________________________
>> Qt-interest mailing list
>> Qt-interest at trolltech.com
>> http://lists.trolltech.com/mailman/listinfo/qt-interest
>>
>> <<--------------------------------------------------------------------------------->> 
>>
>> This E-Mail message has been scanned for viruses
>> and cleared by >>SmartMail<< from Smarter Technology, Inc.
>> <<--------------------------------------------------------------------------------->> 
>>
>>   
>
>