XCFramework. XCode 12 and BigSur issues.

We’ve being using fairly new XCFrameworks starting with our v4 major release of SciChart Mobile SDK with macOS. Since then, we’ve added support for Swift Package Manager as well as Mac Catalyst. Everything was fine until the release of macOS BigSur and XCode 12. With that, we’ve started to receive multiple reports that users were unable to distribute their’s apps through the AppStore.

This article will be solely focused on preparation and build process of XCFramework for distribution. There’s plenty of other articles explaining benefits of using XCFrameworks over plain old traditional fat libraries created explicitly via lipo tool.

If you are not interested in researching and investigation of the problem part, scroll down directly to the Summary.

The Problem

As mentioned above, we’ve started to receive multiple reports that they apps got rejected from AppStore distribution. Some of the issues are listed below:

• issues related to ipatool, while trying to distribute via Ad-Hoc builds

ipatool failed with an exception: #<CmdSpec::NonZeroExitException: $ /Applications/Xcode.app/Contents/Developer/usr/bin/python3 /Applications/Xcode.app/Contents/Developer/usr/bin/bitcode-build-tool -v -t /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin --sdk /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.2.sdk -o /var/folders/vz/b0jj0dms4014y5htfv93svfc0000gn/T/ipatool20201130-52097-13rj9hy/thinned-out/arm64/Payload/iosTest.app/Frameworks/SciChart.framework/SciChart --generate-dsym /var/folders/vz/b0jj0dms4014y5htfv93svfc0000gn/T/ipatool20201130-52097-13rj9hy/thinned-out/arm64/Payload/iosTest.app/Frameworks/SciChart.framework/SciChart.dSYM --strip-swift-symbols /var/folders/vz/b0jj0dms4014y5htfv93svfc0000gn/T/ipatool20201130-52097-13rj9hy/thinned-in/arm64/Payload/iosTest.app/Frameworks/SciChart.framework/SciChart
Status: pid 52150 exit 1
Stdout:
    SDK path: /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/

• direct rejects from the AppStore

Invalid Bundle – One or more dynamic libraries that are referenced by your app are not present in the dylib search path. If your app contains bitcode, bitcode processing may have failed. Because of these errors, this build of your app will not be able to be submitted for review or placed on the App Store.

While first message was apparently an issue with App Thinning, the second one clearly relates to the bitcode, but that wasn’t the case before the macOS and XCode updates. Moreover, not much information was found either in logs or in the similar issues posted on StackOverflow. So we’ve contacted Apple and while waiting were trying to investigate issue on our own.

The Investigation

Since resolving technical issues directly with Apple is known to be super slow and time consuming, we were searching of similar issues over there, as well as trying different combinations of changes in our build settings and build system.

There’s plenty of similar issues reported on StackOverflow, including ours, and most of the workaround is to turn off the bitcode for the distribution, which isn’t suitable in the long-term perspective.

First realistic thing i noticed in the sea of ideas from StackOverflow-like resources was the suggestion that LLVM or GCCinstrumentation might be included in binary, and which will definitely prevent from successful distribution. There’s the Technical Q&A QA1964 which discuss this in more details. We’ve haven’t GCC_INSTRUMENT_PROGRAM_FLOW_ARCS build setting and confirmed from our build logs that the flags -fprofile-instr-generate and -fcoverage-mapping were applied while performing release builds. So we’ve set this to NO and those flags were gone from the compiler logs, but the issue was still persisted.

From here, we’ve started extensive testing using Ad-Hoc builds, investigating logs, which still wasn’t very helpful outputting hundred thousands lines of logs, where most useful was something like this:

-= Output =-
    ld: warning: -sdk_version and -platform_version are not compatible, ignoring -sdk_version
    Undefined symbols for architecture arm64:
      "___llvm_profile_runtime", referenced from:
          __hidden#25821_ in 0832.o
          __hidden#25821_ in 0833.o
          __hidden#25821_ in 0834.o
          __hidden#25821_ in 0835.o
          __hid
2020-11-30 16:19:17 +0000  den#25821_ in 0836.o
          __hidden#25821_ in 0837.o
          __hidden#25821_ in 0838.o
          ...
    ld: symbol(s) not found for architecture arm64
    Exited with 1z
    
    Failed to compile bundle: /var/folders/vz/b0jj0dms4014y5htfv93svfc0000gn/T/SciChart6aun_sfs/SciChart.arm64.xar

Stderr:

    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2808:in `block in CompileOrStripBitcodeInBundle'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2747:in `each'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2747:in `CompileOrStripBitcodeInBundle'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:3016:in `block in ProcessIPA'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2978:in `each'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:2978:in `ProcessIPA'
    from /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool:3928:in `<main>'
2020-11-30 16:19:17 +0000  /Applications/Xcode.app/Contents/Developer/usr/bin/ipatool exited with 1

This only suggest on missing some symbols without clear understanding which ones and where to look for.

The strange part here, is that this issue appears only when including pre-compiled .framework or .xcframework. If we were using our framework .xcodeproj directly, everything was fine and distribution worked as it should be in the first place.

The Solution

At some point, we’ve received response from Apple. They tried to recompile our code from bitcode, and send us the following snippet:

‘/Volumes/Tools/Xcode-1220.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.ios.a’, ‘-o’, ‘/private/var/folders/1w/7908nl_17jg5f4312zkdqp040000gn/T/SciChartf3k2nzqg/SciChart.arm64.out’]n    -= Output =-n    ld: warning: -sdk_version and -platform_version are not compatible, ignoring -sdk_versionn    Undefined symbols for architecture arm64:n      “___llvm_profile_runtime

suggesting that there’s references to libclang_rt and ___llvm_profile_runtime symbols, which indicates, that the library is build with LLVM instrumentation. So turns out that GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = NO and absence of -fprofile-instr-generate and -fcoverage-mapping isn’t enough to remove mentioned LLVM instruments, which are useful and should be used for debugging and testing only.

Apple engineer pointed me back to the Technical Q&A QA1964, and emphasize that i should look for the __llvm_prf as stated in technical document, because he confirms that he did found it, which confirms that the library has LLVM instrumentation included with build.

So we went back, and inspect our library for LLVM instrumentation (as well as GCC) as described in the Listing 3 from previously mentioned article:

otool -l -arch all SciChart | grep __llvm_prf

which provide us with the following output:

sectname __llvm_prf_cnts
sectname __llvm_prf_data
sectname __llvm_prf_names
sectname __llvm_prf_vnds

The above output confirms, that SciChart.framework does contains previously mentioned LLVM instruments. Finally we found the real reason why applications which used SciChart were rejected from store, and has ipatool issues while trying to distribute via Ad-Hoc.

The thing is, that we explicitly were turning that off for our release builds using the following build settings:

CLANG_ENABLE_CODE_COVERAGE = NO
GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = NO

So we contacted Apple and theirs engineer helped us again, pointing out, that for frameworks meant for distribution, we should be using archive builds while preparing XCFramework, release one isn’t enough. In addition libraries which meant to be distributed should also contain SKIP_INSTALL = NO, which is mentioned in the official documentation on how to Create an XCFramework.

The official documentation explains SKIP_INSTALL setting as: «If enabled, don’t install built products even if deployment locations are active». Since we’re developing the framework supposed to be distributed, we thought that it should be set to YES. Turns out we were wrong, and is must be set NO. Short explanation is it causes XCode to follow the deployment paths and output the framework inside the archive, so it can be picked up  for an XCFramework.

Documentation also stands, that BUILD_LIBRARY_FOR_DISTRIBUTION must be set to YES, which we were using already, but since it’s the “must“, I thought it worth mentioning here as well.

Conforming to all of the above prerequisites removed LLVM instruments from the final binary and hence fixes the distribution issues.

Summary

Considering all of the above, I will try to sum up required steps to prepare successful builds which can be used to create XCFrameworks and later on distributed without problems:

• Using archive builds is a must, release build isn’t enough
• BUILD_LIBRARY_FOR_DISTRIBUTION must be set to YES
• SKIP_INSTALL must be set to NO
• GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = NO to turn off GCC instrumentation and remove them from the binary
• CLANG_ENABLE_CODE_COVERAGE = NO to turn off code coverage tools from the binary

Having all of the above helped to solve our preparing and distribution problem and hopefully save you some time if you happened to face same issues as we did.