[Development] templated QObjects [was: Re: We are planning to upgrade qdoc to use clang for parsing C++]

[Thiago Macieira]

On quarta-feira, 2 de março de 2016 20:59:41 PST Milian Wolff wrote:
> Hey Thiago,
> 
> what is "the runtime merging problem on Windows"?

Ever heard of the dynamic_cast problem on Windows? It's the same.

Here's the problem:

QMetaObjects are identified by their pointer addresses: two meta objects are 
the same if their pointer addresses are the same (remember: QMetaObject has no 
operator==). qobject_cast uses that feature to conclude whether a given object 
derives from a given class.

The way it's currently designed in Qt, the meta object is exported 
(Q_DECL_EXPORT) from the DLL in which the class is defined. Obviously, that can 
only happen for concrete types, not for templates. As I explained, we could 
add a feature to allow the class author to export all possible instantiations 
of a template. Each and every instantiation would be exported from the DLL. 
That's, in fact, why a full listing is required: to determine which 
instantiations to export in the first place.

The other suggestion done here is that the user of a template create the meta 
object. That's how typeinfo works for types without virtual tables: each place 
where typeid() is called, the typeinfo is generated, then merged at runtime by 
the dynamic linker. On ELF systems without any special compiler flags, this 
works because symbols are global by default ("default" visibility) and all 
references to any symbol name are accessed via the GOT, which ensures that 
only one copy is active and all accesses get the same pointer address.

Where this breaks down:

1) Windows: the PE-COFF file format does not work like ELF. Symbols are by 
local (the default), __declspec(dllexport), or __declspec(dllimport). If the 
symbol is local or exported, then the compiler generates access assuming that 
the copy in the current DLL is the active one; if it's imported, then the 
compiler generates code assuming it exists in another DLL and will not emit a 
copy.

This means that if a DLL has a copy, it assumes its copy is active. If it has 
no copy, some other DLL must have it. What's more, imports are associated with 
a particular DLL, so the compile-time linker needs to know which DLL contains 
the symbol.

I don't know how or even if throwing template types or dynamic_cast'ing them 
works on Windows. It's possible it doesn't work and will never work. I don't 
care to find out.

2) "hidden" visibility: modern libraries on Unix today compile with 
-fvisibility=hidden -fvisibility-inlines-hidden, like Qt does. Many of them, 
like Qt, heavily pollute the global namespace if you don't use those flags, and 
that's assuming they work at all. Every type with hidden visibility is, as the 
name says, not exported to the ELF dynamic symbol table, which means the 
dynamic linker doesn't see them and will not merge with other copies.

3) -Bsymbolic / "protected" visibility: this instructs the compiler and linker 
that the exported symbols are not subject to preemption and that, like 
Windows's __declspec(dllexport), the copy in this ELF module is always the 
active one and local accesses need not go through the GOT (that's why we use 
it). If this assumption fails, crazy things happen, as we've seen with 
platforms other than x86 for our -Bsymbolic usage.

Summary: this is where the theory of the C++ Standard and reality do not 
agree. When you take the ABIs into consideration, the C++ Standard's definition 
of ODR is just wishful thinking.

In time: C++17 Modules do not solve this issue. Modules replace some use of 
headers and #include; they have nothing to do with DLLs and symbol exporting/
importing.

Glossary:
 * ELF: Executable and Linkable Format, the file format for all object files, 
libraries, executables and core dumps on modern Unix systems, first deployed by 
Sun on Solaris.
 * PE-COFF: Portable Executable COFF, Microsoft's variant of the COFF object 
file format, used for DLLs and executables (not for .obj files, that's OMF).
 * GOT: Global Offset Table, a technique used to achieve position-
independence. Whenever a symbol "foo" is referenced, instead of writing its 
address into the code, the compiler generates an indirect load of the symbol's 
address from a fixed location in the GOT, which the dynamic linker will write 
to once it has finished loading all modules and can resolve all symbols. This 
can be used to make the code read-only and shareable. See also PLT and the GOT 
pointer (which is the value that the PIC register carries).

-- 
Thiago Macieira - thiago.macieira (AT) intel.com
  Software Architect - Intel Open Source Technology Center