[Development] Getting QtWebEngineProcess.app to run in sandbox after being signed

Tue May 9 13:06:50 CEST 2017

> Could you please describe what's the procedure for getting the private
API usage list from the MAS?

Unfortunately I do not know that there is one. I found these API calls only
because when they rejected my app, they sent me a list of the API calls
that they wouldn't allow to be used.

I will spend some time over the weekend with this patch and see if I can
get another version of my app submitted. I'll keep you updated.

On Tue, May 9, 2017 at 5:32 AM, Alexandru Croitor <alexandru.croitor at qt.io>
wrote:

> I took a look at the patches, and I think it should be safe-ish to apply
> them.
>
> That is only the case because the Bootstrap Sandbox which uses the private
> XPC API is not actually used at the moment in Qt WebEngine.
> The other private API patches listed there are either not relevant to
> WebEngine, already handled by WebEngine, or safe enough to remove.
>
> Could you please describe what's the procedure for getting the private API
> usage list from the MAS?
>
> On 08 May 2017, at 22:29, Adalid Claure <aclaure at gmail.com> wrote:
>
> For the fun of it, I tried submitting my app and yeah, it was rejected:
>
> Your app uses or references the following non-public API(s):
>
> private spi symbol(s) in framework: '/usr/lib/libSystem.B.dylib'
> +++                : xpc_dictionary_set_mach_send
> +++                : xpc_mach_send_get_right
>
> I did, however find this: https://github.com/electron/libchromiumcontent/
> tree/master/patches-mas
>
> I may try applying those patches just to see what happens. I'll let you
> know.
>
> Thanks again for your help.
>
>
>
> On Mon, May 8, 2017 at 11:35 AM, Adalid Claure <aclaure at gmail.com> wrote:
>
>> I just meant are the changes to QtWebEngine you suggested stable enough
>> for "production". But if it's still using private API, then I guess there's
>> no point in trying to submit it to MAS.
>>
>> Damn.
>>
>> On Mon, May 8, 2017 at 11:32 AM, Alexandru Croitor <
>> alexandru.croitor at qt.io> wrote:
>>
>>>
>>> On 08 May 2017, at 17:22, Adalid Claure <aclaure at gmail.com> wrote:
>>>
>>> I changed the string like you suggested and it worked!
>>>
>>> Of course, I know this is very hack'ish. How stable do you think this
>>> is?
>>>
>>>
>>> I wouldn't know because this is the first time I tried doing it, and
>>> pretty much nobody tests for this unless they need it.
>>>
>>> I've noticed that the Electron framework (based on Chromium) documents
>>> the whole submit to Mac App Store sandbox / codesign process, so I assume
>>> it worked for some people in some scenario. To what extent, I wouldn't know.
>>>
>>> My app generates all the HTML it displays using QtWebEngine and it's
>>> nothing complicated. Do you think these changes in QtWebEngine I made per
>>> your suggestions are worth submitting to the MAS?
>>>
>>>
>>> I'm not sure what you mean by "submitting the changes to MAS". You will
>>> obviously need the changes to your application, to at least try to submit
>>> to the MAS, but you will probably fail because of the private API functions
>>> that are still used in the code.
>>>
>>>
>>> So far though, the plan is to document for Qt 5.9.0 that Qt WebEngine in
>>> its current state will not be app store compliant (because of the sandbox
>>> issue, and because of the private API usage).
>>>
>>>
>>> On Mon, May 8, 2017 at 7:41 AM, Alexandru Croitor <
>>> alexandru.croitor at qt.io> wrote:
>>>
>>>>
>>>> On 08 May 2017, at 13:33, Adalid Claure <aclaure at gmail.com> wrote:
>>>>
>>>> > Well, to be fair, it seems you haven't heeded my advice, and are
>>>> going with the approach of sandboxing the
>>>> > whole application, instead of leaving that to be done by Chromium.
>>>>
>>>> Hmm.. I'm not sure what you mean. How exactly should I be signing my
>>>> app then? Per the MAS, I have to sign it my app with "
>>>> com.apple.security.app-sandbox".
>>>>
>>>>
>>>> I would have expected that signing without sandboxing should be
>>>> possible. But if it is absolutely required, then that's unfortunate,
>>>> because of the mentioned "unsupported by Chromium" territory.
>>>>
>>>> > Regarding the issues you are seeing now, it is because of the
>>>> "com.apple.security.application-groups"
>>>> > entitlement key. That value (team id + "." + random domain-like
>>>> suffix string) has to be used by Chromium
>>>> > code, so that access is granted to the cross-process resources. By
>>>> default Chromium uses
>>>> > "org.chromium.Chromium" prefix as the application group key, which
>>>> results in the permission denied
>>>> > issues.
>>>>
>>>> I see. I've been using the <teamid>.<app bundle id> entitlement for my
>>>> main app, but I guess Chromium's code ignores that and uses this hard coded
>>>> value?
>>>>
>>>>
>>>> Yes, currently the chromium part of WebEngine ignores the bundle id,
>>>> and just hardcodes it.
>>>>
>>>>
>>>> > You can modify the "BaseBundleID" method in
>>>> "/qt_source/qtwebengine/src/
>>>> > 3rdparty/chromium/base/mac/foundation_util.mm:241" to hardcode the
>>>> bundle id to your "team_id + suffix"
>>>> > entitlement key. That made it work for me.
>>>>
>>>> Ok, I will give this a try.
>>>>
>>>> Thanks again for your help.
>>>>
>>>>
>>>> On Mon, May 8, 2017 at 4:43 AM, Alexandru Croitor <
>>>> alexandru.croitor at qt.io> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Well, to be fair, it seems you haven't heeded my advice, and are going
>>>>> with the approach of sandboxing the whole application, instead of leaving
>>>>> that to be done by Chromium.
>>>>>
>>>>> Regarding the issues you are seeing now, it is because of the
>>>>> "com.apple.security.application-groups" entitlement key. That value
>>>>> (team id + "." + random domain-like suffix string) has to be used by
>>>>> Chromium code, so that access is granted to the cross-process resources. By
>>>>> default Chromium uses "org.chromium.Chromium" prefix as the application
>>>>> group key, which results in the permission denied issues.
>>>>>
>>>>> You can modify the "BaseBundleID" method in
>>>>> "/qt_source/qtwebengine/src/3rdparty/chromium/base/mac/found
>>>>> ation_util.mm:241" to hardcode the bundle id to your "team_id +
>>>>> suffix" entitlement key. That made it work for me.
>>>>>
>>>>> Regards, Alex.
>>>>>
>>>>> On 06 May 2017, at 14:46, Adalid Claure <aclaure at gmail.com> wrote:
>>>>>
>>>>> FWIW, I tried these changes but like you said I ran into another error
>>>>> (though, I'm pretty sure it's one I've seen before).
>>>>>
>>>>> 5/6/17 08:35:40.000 kernel[0]: Sandbox: QtWebEngineProce(5958) deny(1)
>>>>> mach-lookup org.chromium.Chromium.rohitfork.5957
>>>>> 5/6/17 08:35:40.000 kernel[0]: Sandbox: QtWebEngineProce(5959) deny(1)
>>>>> mach-lookup org.chromium.Chromium.rohitfork.5957
>>>>> 5/6/17 08:35:40.070 QtWebEngineProcess[5958]: [0506/083540:ERROR:
>>>>> mach_port_broker.mm(43)] bootstrap_look_up: Permission denied (1100)
>>>>> 5/6/17 08:35:40.070 QtWebEngineProcess[5959]: [0506/083540:ERROR:
>>>>> mach_port_broker.mm(43)] bootstrap_look_up: Permission denied (1100)
>>>>>
>>>>> Does this look like the other issues you were referring to?
>>>>>
>>>>> There is progress in the sense that the QtWebEngine widget is now
>>>>> white, instead of invisible (since it wasn't loading, I presume) like
>>>>> before. Of course, nothing renders inside it though.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 4, 2017 at 9:32 AM, Alexandru Croitor <
>>>>> alexandru.croitor at qt.io> wrote:
>>>>>
>>>>>> It's in the function SandboxCompiler::CompileAndApplyProfile in
>>>>>> qt59_source/qtwebengine/src/3rdparty/chromium/content/common/
>>>>>> sandbox_mac.mm:150
>>>>>> the whole if branch.
>>>>>>
>>>>>> But please do note my warning, even if it appears to work, it does
>>>>>> not mean you will not have issues. Chromium has its own special sandbox
>>>>>> profile (set of rules) that it applies to the subprocess, and it never
>>>>>> applies it to the main process. So by simply enabling default sandboxing
>>>>>> for both, you invite untested territories.
>>>>>>
>>>>>>
>>>>>> On 04 May 2017, at 15:26, Adalid Claure <aclaure at gmail.com> wrote:
>>>>>>
>>>>>> I am the one who logged the bug.
>>>>>>
>>>>>> Can you point me to the line of code that you commented out where you
>>>>>> were able to get it to work?
>>>>>>
>>>>>> On Thu, May 4, 2017 at 9:15 AM, Alexandru Croitor <
>>>>>> alexandru.croitor at qt.io> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm actually looking into this at the moment, as per the created
>>>>>>> issue here QTBUG-60443
>>>>>>>
>>>>>>> I haven't completely finished my investigation, which I will post to
>>>>>>> the bug report.
>>>>>>>
>>>>>>> But the gist of it is:
>>>>>>> - the WebEngine / Chromium sub process calls manually sandbox_init,
>>>>>>> which is the macOS private API for enabling the sandbox. That interferes
>>>>>>> with the sandbox which you enable via codesign entitlements, which
>>>>>>> gives the "sandbox reinit denied' issue, and crashed process.
>>>>>>>
>>>>>>> - if the above call is commented out, then you need to make sure
>>>>>>> that the Qt Application is signed with sandbox setup entitlements, and the
>>>>>>> WebEngineProcess is signed with sandbox "inherit" entitlements, otherwise I
>>>>>>> couldn't get it to  work.
>>>>>>>
>>>>>>> - Both of those entitlements have to specify a valid Team ID +
>>>>>>> prefix in the entitlment com.apple.security.application-groups
>>>>>>> key.
>>>>>>>
>>>>>>> - Google Chrome itself does not enable sandboxing via codesign
>>>>>>> entitlements, but only enables it for subprocess via sandbox_init, leaving
>>>>>>> the main process without a sandbox.
>>>>>>>
>>>>>>> Thus my preliminary suggestion is not to enable codesign sandboxing
>>>>>>> for QtWebengine applications.
>>>>>>>
>>>>>>> I'm open to discussion on this though.
>>>>>>>
>>>>>>> Note that the private API use inside WebEngine, which is removed via
>>>>>>> appstore_compliant configure switch, is a separate issue from sandboxing.
>>>>>>>
>>>>>>> Regards, Alex.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 04 May 2017, at 14:45, Morten Sørvig <Morten.Sorvig at qt.io> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Not sure if I can be of much help, but:
>>>>>>>
>>>>>>> - This thread discusses and solves a similar problem:
>>>>>>> https://forum.qt.io/topic/49250/solved-qtwebengineprocess-no
>>>>>>> t-working-in-sandboxed-application
>>>>>>>
>>>>>>> - If this could be reduced to a simple sandboxed-app-with-helper-process
>>>>>>> test case (no QtWebEngine usage), that that’s something I could look at,
>>>>>>> and something we could eventually add an autotest for.
>>>>>>>
>>>>>>>
>>>>>>> Morten
>>>>>>>
>>>>>>>
>>>>>>> On 28 Apr 2017, at 18:49, Adalid Claure <aclaure at gmail.com> wrote:
>>>>>>>
>>>>>>> I have a desktop app that I have been trying to get onto the Mac App
>>>>>>> store but I have been having problems getting it to run in sandbox mode.
>>>>>>> For context I am (preferably) using Qt 5.8 running on macOS 10.11.6.
>>>>>>>
>>>>>>> The crux seems to be QtWebEngineProcess.app refuses to run after I
>>>>>>> codesign the bundle. As a result, my QtWebEngine component doesn't load. I
>>>>>>> am using this QtWebEngine component as part of my app's UI.
>>>>>>>
>>>>>>> When the app starts I see the following errors in Console:
>>>>>>>
>>>>>>> kernel[0]: Sandbox: QtWebEngineProce(20764) deny(1) mach-lookup
>>>>>>> org.chromium.Chromium.rohitfork.20763
>>>>>>> kernel[0]: Sandbox: QtWebEngineProce(20765) deny(1) mach-lookup
>>>>>>> org.chromium.Chromium.rohitfork.20763
>>>>>>> QtWebEngineProcess[20764]: [0427/071053:ERROR:mach_broker_mac.mm(52)]
>>>>>>> bootstrap_look_up: Permission denied (1100)
>>>>>>> QtWebEngineProcess[20765]: [0427/071053:ERROR:mach_broker_mac.mm(52)]
>>>>>>> bootstrap_look_up: Permission denied (1100)
>>>>>>> kernel[0]: Sandbox: QtWebEngineProce(20764) deny(1)
>>>>>>> forbidden-sandbox-reinit
>>>>>>>
>>>>>>> My build process is pretty straight forward:
>>>>>>>
>>>>>>> 1. Run macdeployqt on the app, using the -appstore-compliant.
>>>>>>> 2. Sign all of the Qt Frameworks and PlugIns individually with my
>>>>>>> app's entitlement file.
>>>>>>> 3. Sign QtWebEngineProcess.app with the following entitlements file:
>>>>>>>
>>>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>>>> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "
>>>>>>> http://www.apple.com/DTDs/PropertyList-1.0.dtd">
>>>>>>> <plist version="1.0">
>>>>>>> <dict>
>>>>>>>    <key>com.apple.security.app-sandbox</key>
>>>>>>>    <true/>
>>>>>>>    <key>com.apple.security.inherit</key>
>>>>>>>    <true/>
>>>>>>> </dict>
>>>>>>> </plist>
>>>>>>>
>>>>>>> 4. Call codesign on the overall MyProgram.app bundle with the
>>>>>>> entitlements file from Step 2.
>>>>>>>
>>>>>>> I have tried numerous things all in combination with one another,
>>>>>>> including:
>>>>>>>
>>>>>>> a. built QtWebEngine using WEBENGINE_CONFIG+=use_appstore_compliant_code
>>>>>>> (per the notes here: https://doc.qt.io/qt-5/qtweben
>>>>>>> gine-platform-notes.html#mac-app-store-compatibility)
>>>>>>> b. use macdeployqt's -codesign, even though the binarys have to be
>>>>>>> signed a second time after this in order to apply the entitlements
>>>>>>> c. sign QtWebEngineProcess.app with CFBundleIdentifier equal to
>>>>>>> 'com.qt-project.Qt.QtWebEngineProcess' and with my own app's bundle
>>>>>>> ID.
>>>>>>> d. tried linking with Qt 5.7
>>>>>>> e. tried linking with Qt 5.6.2 which *did* run but then gets
>>>>>>> rejected by Apple because:
>>>>>>>
>>>>>>> -------------------------------
>>>>>>> Your app uses or references the following non-public API(s):
>>>>>>>
>>>>>>> framework: '/System/Library/Frameworks/Ap
>>>>>>> pKit.framework/Versions/C/AppKit'
>>>>>>> : NSAccessibilityUnregisterUniqueIdForUIElement
>>>>>>> : _NSAppendToKillRing
>>>>>>> : _NSDrawCarbonThemeBezel
>>>>>>> : _NSDrawCarbonThemeListBox
>>>>>>> : _NSInitializeKillRing
>>>>>>> : _NSNewKillRingSequence
>>>>>>> : _NSPrependToKillRing
>>>>>>> : _NSSetKillRingToYankedState
>>>>>>> : _NSYankFromKillRing
>>>>>>>
>>>>>>> framework: '/System/Library/Frameworks/Ap
>>>>>>> plicationServices.framework/Versions/A/ApplicationServices'
>>>>>>> : CGSSetDenyWindowServerConnections
>>>>>>> : CGSShutdownServerConnections
>>>>>>> : CTFontCopyDefaultCascadeList
>>>>>>>
>>>>>>> The use of non-public APIs is not permitted on the App Store as it
>>>>>>> can lead to a poor user experience should these APIs change.
>>>>>>> -------------------------------
>>>>>>>
>>>>>>> I have chronicled a lot of this in this thread here (
>>>>>>> https://forum.qt.io/topic/78518/sandbox-app-for-the-mac-app
>>>>>>> -store-with-qt-5-8-and-qtwebengineprocess) but the problem
>>>>>>> persists.
>>>>>>>
>>>>>>> Does anyone have any suggestions? Does anyone know of any apps on
>>>>>>> the Mac App Store that use QtWebEngine?
>>>>>>>
>>>>>>> Thanks.
>>>>>>> _______________________________________________
>>>>>>> Development mailing list
>>>>>>> Development at qt-project.org
>>>>>>> http://lists.qt-project.org/mailman/listinfo/development
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Development mailing list
>>>>>>> Development at qt-project.org
>>>>>>> http://lists.qt-project.org/mailman/listinfo/development
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.qt-project.org/pipermail/development/attachments/20170509/fe7c9246/attachment.html>