[Development] Qt 4.x and Qt 5 frameworks should use @rpath (QTBUG-31814)

Ziller Eike Eike.Ziller at digia.com
Mon Aug 11 16:42:08 CEST 2014


On Aug 3, 2014, at 12:39 AM, Jake Petroules <jake.petroules at petroules.com> wrote:

> On 2014-08-02, at 05:04 AM, Adam Strzelecki <ono at java.pl> wrote:
> 
>>> Please don't exaggerate the performance implications of copying a few frameworks on the first build (…)
>> 
>> This can be done, but discussion is going off-topic.
>> 
>> Let us try first to move Qt frameworks to use @rpath & remove some unnecessary operations while keeping current workflow with minimal changes to the code. Therefore I hereby propose to (1) keep rpath to Qt SDK in built app executable, (2) not copy anything on build, (3) make macdeploy modify ONLY executable removing absolute rpath (instead fixing frameworks), and (4) just copy needed frameworks and (5) optionally sign whole bundle.
>> 
>> This will not require any wrappers, environment variables or changes in Qt Creator to run your app. Only Qmake and macdeploy need to be changed. 
>> 
>> Frankly I don't see anything wrong with keeping full path to SDK in executable when it isn't yet completely bundled. And the initial intention was to NOT touch Qt SDK modules once they are built.
>> 
>> Once this is done & working we can follow this discussion and think what can be done next.
>> 
>> -- 
>> Adam
> 
> This isn't off topic at all. @rpath, copying frameworks, DYLD_ environment variables are all inseparably linked and very much part of the same discussion and overall issue. Your proposal to simply add @rpath and do nothing else has no benefits. What problem does it solve, other than deleting a bit of code from macdeployqt that currently works and will continue to work without maintenance? None. There is no point in doing it unless we go all the way and make the Qt SDK completely relocatable.
> 
> I understand you want to start by completing one objective at a time, but keep in mind each of these objectives is part of a greater overall goal. Alone, they are pointless.
> 
>>> Actually you CAN add Info.plist to bundle-less apps using the linker arguments: `-sectcreate __TEXT __info_plist Info.plist` which will embed it in the binary. All OS X APIs handle this transparently. Though, LSEnvironment is not a good solution for the problem we are trying to solve here.
>> 
>> I was expecting that ;) But wasn't your intention to not modify binary once it has been built and not hardcode any paths into it?
> 
> Yes, which is why I said LSEnvironment is not a good solution.
> 
>>> Why would you do any of these things?
>> 
>> Because you dislike adding absolute rpath to Qt SDK in built binary during dev process.
> 
> All of those things involve hardcoding absolute paths *somewhere*. There should be no absolute paths *anywhere* except environment variables that are automatically set per-session.
> 
>>> "Inject"? DYLD_LIBRARY_PATH is an environment variable. You simply set it in the process's environment before spawning.
>> 
>> It is not so simple, first of all as you shown it works only from console, but if you launch it via GUI you need this variable to be set in launchd. As ~/.launchd.conf doesn't work anymore since Lion only permanent solution is to use /etc/launchd-user.conf which require admin to create/modify.
>> 
>> Secondly DYLD_LIBRARY_PATH has strong security implications as the path is searched BEFORE default locations. So it does in fact let you inject/replace libraries. This is the reason dyld disables that for any process running under root account.
>> 
>> man dyld
>> 
>> DYLD_LIBRARY_PATH
>> 
>> 	The dynamic linker searches these directories BEFORE it searches the default locations for libraries.
>> 
>> What might be considered there is DYLD_FALLBACK_FRAMEWORK_PATH.
> 
> This is another excuse - there are no real security implications in practice. If you're even thinking about this, your machine has already been compromised and you no longer own it. Remember that this is a development time concern, too, and has nothing to do with a production environment.
> 
> The security conscious Apple Inc. doesn't seem to think it's a big problem either, because Xcode sets it during development (see below).
> 
> I'm curious in what realistic scenario you think this is a real problem.
> 
>> 
>>> In a terminal session, simply:
>>> 
>>> $ export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/path/to/Qt/Frameworks
>>> $ open /apps/MyApp.app
>>> 
>>> In Creator, just click run and this will be handled automatically. In qbs just type `qbs run -p foo` and this will be handled automatically.
>> 
>> This won't work for apps launched directly via Finder/Dock.
> 
> Like I said, that doesn't matter. It didn't work in Qt 4 and the fact that it works in Qt 5 is a mere side effect. No one will notice.

Qt 4 worked exactly like Qt 5 does in this regard.

> Most people build Qt apps on OS X using some IDE, most likely Qt Creator. Are you seriously telling me you click build, go find the build directory, and launch the app in Finder?

I very often start my developed app (Qt Creator) from Finder/Spotlight/Launchbar. For example when I want to show someone something, and I currently do not have Qt Creator open, or have Qt Creator open with a different application or branch of the application.

> That's ridiculous. You click the run button. If you're willing to go through all the former effort you can easily export DYLD_LIBRARY_PATH at the beginning of your terminal session and type `open MyApp.app`. Note that the open command uses launch services which is the same as double clicking the bundle in Finder.

I’m pretty sure that

export DYLD_LIBRARY_PATH=......
open Foo.app

does NOT work. “open” runs the application in the launchd environment.

> 
> Furthermore, if you simply copy frameworks into the bundle at build time like every other native OS X app on the planet, this becomes a non-issue and you don't even need DYLD_LIBRARY_PATH. With my QMAKE_EMBEDDED_FRAMEWORKS suggestion, too, there would be a choice between the two, and be closer to native tools behaviour at the same time.

Copying the Qt frameworks at build time possibly would be the best thing to do (and is probably what you do when using some “native” OS X frameworks out there). If we’d start doing that, we’d want to only copy the frameworks that are actually used though. And the Qt plugins and Qt Quick imports, and only the ones that are needed/wanted. I don’t see how one could argue about doing that for OS X target only though, and in the end it would mean putting all the functionality of (mac/win/android)deployqt into the build process. That would definitely be more than just adapting some rpaths/ids set in the build process and install_name call adaptions in macdeployqt.

>From the Qt + Qt Creator developer point of view it would be a bit ugly when that is done in a development build, because updating my Qt then means that I have to rebuild my Qt Creator versions to see the effects, but that could be argued to be a side-case.

> ANY solution that involves placing absolute paths in ANY file inside the MyApp.app/ directory tree is a wrong solution.
>> 
>> You are wrong there. All system libs are referenced with their absolute paths.
> 
> Those are system libraries that are part of the operating system, present on every OS X installation in the world and not designed to be relocatable. Qt is not a system library and should be relocatable. Anything residing in /System has no bearing on this discussion.
> 
>> Moreover the Qt's lib/ absolute path will be there only during development, in deployed binary absolute path will be removed via install_name_tool -delete_rpath.
> 
> This still breaks the rule of absolute paths appearing somewhere. We do not want absolute paths anywhere during development except in per-session environment variables.
> 
>>> Note that Qt Creator already does the logical equivalent of what I described on Windows; the Qt libraries path is added to the PATH when the process is launched. Launching directly in Explorer doesn't work, and launching directly from Finder doesn't need to work either.
>> 
>> Currently it DOES WORK when launching via Finder, because it does hardcode absolute path, so I don't see any reason it should stop working. Also it does work like that on Linux too.
> 
> Linux is not OS X (also did you forget about $ORIGIN?). And if we're going to compare to other operating systems, currently it DOESN'T work on Windows. Never seen a complaint about it.

We regularly have confused Windows developers on the Qt Creator IRC/mailing list, because their apps do not start from Explorer.
That everyone is used to Windows development being inconvenient shouldn’t be a reason to introduce that on the other platforms too ;)

> And that is not a problem because in 2014 people use IDEs and click the run button which configures search paths for you. This is why we have tools.
> 
>>> This is closer to how the native tools do things anyways.
>> 
>> Which tools? AFAIK Xcode doesn't copy external frameworks into app bundle by default.
> 
> Yes it does. It has much specific functionality and behaviours for doing this and this technique is encouraged by Apple and used by pretty much every Xcode-using project on the planet. And the copying happens at development time, for both debug and release. No one postpones this process to a release-time packaging step like macdeployqt.
> 
> Examples of how Xcode facilitates this:
> 
> (1) Embedded binaries chooser on the Info tab of a target configuration
> (2) Copy phase with an explicit selection for Frameworks on the Build Phases tab of a target configuration
> (3) Default prompt to embed a framework in an application target when creating a new framework
> (4) DYLIB_INSTALL_NAME_BASE set to @rpath by default when creating a new framework
> (5) OS X: LD_RUNPATH_SEARCH_PATHS set to @executable_path/../Frameworks for apps and [@executable_path/../Frameworks, @loader_path/Frameworks] for frameworks when creating a new target
> (6) iOS: LD_RUNPATH_SEARCH_PATHS set to @executable_path/Frameworks for apps and [@executable_path/Frameworks, @loader_path/Frameworks] for frameworks when creating a new target
> (7) https://developer.apple.com/library/mac/documentation/macosx/conceptual/BPFrameworks/Tasks/CreatingFrameworks.html (some parts are outdated as DYLD_LIBRARY_PATH is now set automatically, but the rest of the documentation pretty much instructs users to do exactly what I am suggesting)
> 
> This is not a novel idea I just came up with. This has been the standard workflow on OS X for over a decade.
> 
>>> Symlinks make no sense. Just copy the full frameworks, there is no disadvantage to this. This is the standard workflow for development on Apple platforms anyways and has been since NeXTSTEP.
>> 
>> See above.
>> 
>>> I'm sure someone will bring up "performance!!1" but that is a poor argument;
>> 
>> This is going off-topic. It was about to bring @rpath to Qt frameworks not rework entire workflow.
> 
> The whole point of @rpath is to enable this improvement in workflow. By itself there is no point in adding it other than *slightly* simplifying some macdeployqt code.
> 
>>>> I don't think you should add these by default since Qt doesn't need them, so unlikely your app need them. If you use some 2rd party library you are free to extend QMAKE_RPATH list yourself.
>>> 
>>> It might need them. Depends if we change macdeployqt to place a dylibs (non-frameworks) build of Qt libraries inside Libraries instead of Frameworks. Not a major part of the discussion anyways, just an idea.
>> 
>> I don't see point of handling some custom scenarios that anyway are not what is expected (standard) app bundle.
> 
> Fair enough, let's focus on the task at hand.
> 
>>> Let's use standard solutions instead of strange contraptions. Standard solutions involve @rpath along with DYLD_LIBRARY_PATH, copying frameworks at build time, or both (preferably both).
>> 
>> I am sorry, but you are wrong. DYLD_LIBRARY_PATH is nowhere used in Xcode and DYLD_ variables exist for dyld debug purposes.
> 
> Apologies for parroting, but... you are in fact wrong. I've JUST tested this and Xcode does in fact set many environment variables when running an application. I've added NSLog(@"%@", [[NSProcessInfo processInfo] environment]); in my application delegate. Here's a subset of the output:
> 
> 2014-08-02 17:12:48.775 Silverlock[24258:303] {
>     "DYLD_FRAMEWORK_PATH" = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
>     "DYLD_LIBRARY_PATH" = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
>     PATH = "/Applications/Xcode.app/Contents/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin";
>     PWD = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
>     "__XCODE_BUILT_PRODUCTS_DIR_PATHS" = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
>     "__XPC_DYLD_FRAMEWORK_PATH" = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
>     "__XPC_DYLD_LIBRARY_PATH" = "/Users/jakepetroules/Library/Developer/Xcode/DerivedData/Silverlock-dozbffgewlkgfucjgrjfbjopeyhq/Build/Products/DebugAppStore";
> }
> 
> I work with Xcode on a daily basis building native Cocoa apps. I can assure you I'm well aware of the standards and conventions in use by Apple and their development tools. Bringing Qt in line with standard expectations for OS X and iOS development is important to maintain and grow its competitive edge. Tools like macdeployqt are an extremely foreign concept on OS X and iOS and are confusing to newcomers and annoying for veterans. No other SDK that I know if has anything of the sort, nor uses absolute sonames.
> 
> No need to mention MacPorts, Homebrew, etc., either, which probably use absolute paths for most builds. Package management systems in general have a whole set of conventions. For example, Fedora prohibits rpath usage in the package management system. This is correct and justified. Installed packages in a PMS are never relocated. Application bundles and SDKs on OS X *are*.
> 
>> Try to create simple framework and see what happens, checkout install name of built framework.
>> 
>> $ otool -L /Users/ono/Library/Developer/Xcode/DerivedData/TestFramework-aduwyulqilsereeqgoizoxubgbdy/Build/Products/Debug/TestFramework.framework/TestFramework 
>> /Users/ono/Library/Developer/Xcode/DerivedData/TestFramework-aduwyulqilsereeqgoizoxubgbdy/Build/Products/Debug/TestFramework.framework/TestFramework:
>> 	/Library/Frameworks/TestFramework.framework/Versions/A/TestFramework (compatibility version 1.0.0, current version 1.0.0)
>> 	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
>> 	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1056.13.0)
>> 	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
>> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
>> 
>> 
>> 
>> Now create a Cocoa app and framework from the framework project as see how the framework is referred and check if you can find the framework inside bundle.
>> 
>> $ otool -L /Users/ono/Library/Developer/Xcode/DerivedData/SampleApp-fwmalbohyjlyqgdhjvnicwjqqmfr/Build/Products/Debug/SampleApp.app/Contents/MacOS/SampleApp 
>> /Users/ono/Library/Developer/Xcode/DerivedData/SampleApp-fwmalbohyjlyqgdhjvnicwjqqmfr/Build/Products/Debug/SampleApp.app/Contents/MacOS/SampleApp:
>> 	/Library/Frameworks/TestFramework.framework/Versions/A/TestFramework (compatibility version 1.0.0, current version 1.0.0)
>> 	/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
>> 	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1056.13.0)
>> 	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
>> 	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
>> 	/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1265.19.0)
> 
> Point being? The system default DYLIB_INSTALL_NAME_BASE for frameworks in the Xcode build system happens to be /Library/Frameworks. For new projects it should be set to @rpath (and the IDE does this when creating a new target). Nothing to see here.
> 
>> -- 
>> Adam
> 
> So, let's certainly start with adding @rpath to Qt libraries. But let's not stop there, either.
> -- 
> Jake Petroules - jake.petroules at petroules.com
> Chief Technology Officer - Petroules Corporation
> _______________________________________________
> Development mailing list
> Development at qt-project.org
> http://lists.qt-project.org/mailman/listinfo/development

-- 
Eike Ziller, Senior Software Engineer - Digia, Qt
 
Digia Germany GmbH, Rudower Chaussee 13, D-12489 Berlin
Geschäftsführer: Mika Pälsi, Juha Varelius, Tuula Haataja
Sitz der Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B




More information about the Development mailing list