vcpkg源码之install流程

赵裕(vimerzhao)

2024-06-17 (最后修改: 2024-06-17)

本文约 3000 字，阅读需 6 分钟。

简介

vcpck install的主要工作是"构建(如果需要)和安装依赖包"，是vcpkg中最关键、也是使用者感知最明显的环节，有必要对其流程有一个更底层的理解，这样对一些问题的理解也将更加深刻。

我们首次执行Cmake时，都会有如下信息输出：

-- Running vcpkg install
Fetching registry information from https:/github.com/microsoft/vcpkg.git (HEAD)...
The following packages will be built and installed:
  * abseil[core,cxx17]:arm-neon-android@20230125.3 -- /root/.cache/vcpkg/registries/git-trees/6a337fa251c0ac4489d9c0ea1e2f1c9a7d019eb5
  * brotli:arm-neon-android@1.1.0#1 -- /root/.cache/vcpkg/registries/git-trees/4e5b5ae1ad26c80535c893cc0307121f0398549e
  * bzip2[core,tool]:arm-neon-android@1.0.8#5 -- /root/.cache/vcpkg/registries/git-trees/92e9a8bbf1abbd89872b48ad82fcf75852de1006
....... #省略
Installing 44/45 curl[brotli,c-ares,core,http2,non-http,openssl,ssl,zstd]:arm-neon-android@8.8.0...
Elapsed time to handle curl:arm-neon-android: 2.8 ms
curl:arm-neon-android package ABI: 3adff4b03e8cc179c90168489827352fdd288e42ba831a4a495629d62cb19339
Installing 45/45 tdf:arm-neon-android@2.0...
Building tdf:arm-neon-android@2.0...
/path/to/./ports/tdf: info: installing overlay port from here
-- Using cached /path/to/.vcpkg/downloads/tdf-ca375a42c30097decb1e697c5976d8adaa226003.tar.gz
-- Cleaning sources at /path/to/.vcpkg/buildtrees/tdf/src/adaa226003-e7137e7c2f.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /path/to/.vcpkg/downloads/tdf-ca375a42c30097decb1e697c5976d8adaa226003.tar.gz
-- Using source at /path/to/.vcpkg/buildtrees/tdf/src/adaa226003-e7137e7c2f.clean
-- Configuring arm-neon-android
-- Building arm-neon-android-dbg
-- Building arm-neon-android-rel
-- Performing post-build validation
warning: The software license must be available at ${CURRENT_PACKAGES_DIR}/share/tdf/copyright
error: Found 1 post-build check problem(s). To submit these ports to curated catalogs, please first correct the portfile: /path/to/./ports/tdf/portfile.cmake
Stored binaries in 1 destinations in 31 s.
Elapsed time to handle tdf:arm-neon-android: 3 min
tdf:arm-neon-android package ABI: 1f4f175b2979f99255cbaa5e4c4e2f09f4e0e954bbee81d5e73cbd834c018d85
Total install time: 3 min

....... #省略
-- Running vcpkg install - done

以上，就是vcpkg通过解析vcpkg.json/vcpkg-configuration.json等文件，安装指定版本依赖的过程。

安装的过程涉及的细节比较多，比如:Abi的计算、云缓存的拉取/上传，没有缓存时触发本地构建等。

其中，最值得了解的是：没有命中缓存时，vcpkg如何触发这个port(即三方库)的构建。这是最容易出问题的地方，比如，我们在项目CMake中定义了一个变量，为什么在自定义的portfile.cmake无法访问（最典型的就是明明指定了CMAKE_BUILD_TYPE=Debug，但vcpkg仍会同时构建Debug/Release制品）。

接下来，我们拆解这个过程。

Step1: 主工程CMake阶段

结合官方文档：Install and use packages with CMake | Microsoft Learn，可以发现，CMake使用vcpkg最关键的环节就是指定CMAKE_TOOLCHAIN_FILE为"$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake"。那么，这个cmake其实就是关键的入口。

vcpkg.cmake有近1000行，其核心逻辑之一（即vcpkg install，另一个是find_package的实现）如下(VNote是我加注释)：

# VNote: 前面省略，前置条件检查
if(VCPKG_MANIFEST_MODE AND VCPKG_MANIFEST_INSTALL AND NOT Z_VCPKG_CMAKE_IN_TRY_COMPILE AND NOT Z_VCPKG_HAS_FATAL_ERROR)
    if(NOT EXISTS "${Z_VCPKG_EXECUTABLE}" AND NOT Z_VCPKG_HAS_FATAL_ERROR)
      # VNote: 下载vcpkg-tool，省略
    endif()

    if(NOT Z_VCPKG_HAS_FATAL_ERROR)
        # VNote: 对应上面的日志输出
        message(STATUS "Running vcpkg install")

        # VNote: 开始组装各种参数
        set(Z_VCPKG_ADDITIONAL_MANIFEST_PARAMS)
        if(DEFINED VCPKG_HOST_TRIPLET AND NOT VCPKG_HOST_TRIPLET STREQUAL "")
            list(APPEND Z_VCPKG_ADDITIONAL_MANIFEST_PARAMS "--host-triplet=${VCPKG_HOST_TRIPLET}")
        endif()
        if(VCPKG_OVERLAY_PORTS)
            foreach(Z_VCPKG_OVERLAY_PORT IN LISTS VCPKG_OVERLAY_PORTS)
                list(APPEND Z_VCPKG_ADDITIONAL_MANIFEST_PARAMS "--overlay-ports=${Z_VCPKG_OVERLAY_PORT}")
            endforeach()
        endif()
        if(VCPKG_OVERLAY_TRIPLETS)
        # VNote: 省略一些....

        # VNote: 开始执行vcpkg install这个命令
        execute_process(
            COMMAND "${Z_VCPKG_EXECUTABLE}" install
                --triplet "${VCPKG_TARGET_TRIPLET}"
                --vcpkg-root "${Z_VCPKG_ROOT_DIR}"
                "--x-wait-for-lock"
                "--x-manifest-root=${VCPKG_MANIFEST_DIR}"
                "--x-install-root=${_VCPKG_INSTALLED_DIR}"
                ${Z_VCPKG_FEATURE_FLAGS}
                ${Z_VCPKG_ADDITIONAL_MANIFEST_PARAMS}
                ${VCPKG_INSTALL_OPTIONS}
        # VNote: 省略一些不关键的代码....

        if(Z_VCPKG_MANIFEST_INSTALL_RESULT EQUAL "0")
            # VNote: 对应上面的日志输出
            message(STATUS "Running vcpkg install - done")
            set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS
                "${VCPKG_MANIFEST_DIR}/vcpkg.json")
            if(EXISTS "${VCPKG_MANIFEST_DIR}/vcpkg-configuration.json")
                set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS
                    "${VCPKG_MANIFEST_DIR}/vcpkg-configuration.json")
            endif()
        else()
            message(STATUS "Running vcpkg install - failed")
            z_vcpkg_add_fatal_error("vcpkg install failed. See logs for more information: ${Z_NATIVE_VCPKG_MANIFEST_INSTALL_LOGFILE}")
        endif()
    endif()
endif()

关键逻辑已经在CMake脚本中注明，其实CMake的作用主要是作为vcpkg-tool的Launcher，组装其需要的参数，接下来，我们就分析下vcpkg-tool的逻辑。

Step2: vcpkg-tool阶段

microsoft/vcpkg中主要是各种CMake脚本，其作用有二：

组装参数并调用microsoft/vcpkg-tool(即step1)
配合CMake进行构建（即step3）

vcpkg-tool主要是C++代码实现，负责执行往逻辑比较重的工作，比如：

分析某个依赖的各种间接依赖和版本
网络请求与下载（Binary Cache）
ABI的计算
等等….

这些逻辑并不适合用Cmake处理，其细节也比较多，这里我会通过--debug参数额外获取一些日志，帮助我们更直观地理解vcpkg-tool的工作过程。

Step2.1 核心调用栈

# 构建没有缓存的port
vcpkg::do_build_package(...) (vcpkg-tool\src\vcpkg\commands.build.cpp:994)
vcpkg::do_build_package_and_clean_buildtrees(...) (vcpkg-tool\src\vcpkg\commands.build.cpp:1083)
vcpkg::build_package(...) (vcpkg-tool\src\vcpkg\commands.build.cpp:1426)
# install首先会尝试缓存，没有缓存则进行构建
vcpkg::perform_install_plan_action(...) (vcpkg-tool\src\vcpkg\commands.install.cpp:361)
vcpkg::install_execute_plan(...) (vcpkg-tool\src\vcpkg\commands.install.cpp:604)
vcpkg::command_install_and_exit(...) (vcpkg-tool\src\vcpkg\commands.install.cpp:1365)
(anonymous namespace)::inner(...) (vcpkg-tool\src\vcpkg.cpp:147)
# vcpkg-tool的入口
main (vcpkg-tool\src\vcpkg.cpp:400)

Step2.2 触发构建

C++的流程不用展开分析，因为细节太多，其本质还是组装出一个构建port制品的命令，以vcpkg install tracy为例，如果没有缓存，则会触发其构建，vcpkg-tool本身没有构建能力，如果要构建一个port，还要重新调回vcpkg中的CMake脚本，对应日志如下：

[DEBUG] 1011: execute_process(GIT_CEILING_DIRECTORIES=/data/research/vcpkg/buildtrees  /data/research/vcpkg/downloads/tools/cmake-3.27.
1-linux/cmake-3.27.1-linux-x86_64/bin/cmake "-DALL_FEATURES=cli-tools;crash-handler;gui-tools;on-demand;" -DCURRENT_PORT_DIR=/data/rese
arch/vcpkg/ports/tracy -D_HOST_TRIPLET=x64-linux "-DFEATURES=core;crash-handler" -DPORT=tracy -DVERSION=0.10.0 -DVCPKG_USE_HEAD_VERSION
=0 -D_VCPKG_DOWNLOAD_TOOL=BUILT_IN -D_VCPKG_EDITABLE=0 -D_VCPKG_NO_DOWNLOADS=0 -DZ_VCPKG_CHAINLOAD_TOOLCHAIN_FILE=/data/research/vcpkg/
scripts/toolchains/linux.cmake -DCMD=BUILD -DDOWNLOADS=/data/research/vcpkg/downloads -DTARGET_TRIPLET=x64-linux -DTARGET_TRIPLET_FILE=
/data/research/vcpkg/triplets/x64-linux.cmake -DVCPKG_BASE_VERSION=2999-12-31 -DVCPKG_CONCURRENCY=17 -DVCPKG_PLATFORM_TOOLSET=external
-DGIT=/usr/bin/git "-DVCPKG_PORT_CONFIGS=/data/research/vcpkg/installed/x64-linux/share/vcpkg-cmake/vcpkg-port-config.cmake;/data/resea
rch/vcpkg/installed/x64-linux/share/vcpkg-cmake-config/vcpkg-port-config.cmake" -DVCPKG_ROOT_DIR=/data/research/vcpkg -DPACKAGES_DIR=/d
ata/research/vcpkg/packages -DBUILDTREES_DIR=/data/research/vcpkg/buildtrees -D_VCPKG_INSTALLED_DIR=/data/research/vcpkg/installed -DDO
WNLOADS=/data/research/vcpkg/downloads -DVCPKG_MANIFEST_INSTALL=OFF -P /data/research/vcpkg/scripts/ports.cmake)
-- TEst
-- Using cached wolfpld-tracy-v0.10.tar.gz.
-- Cleaning sources at /data/research/vcpkg/buildtrees/tracy/src/v0.10-b0e1fd57f2.clean. Use --editable to skip cleaning for the packag
es you specify.
-- Extracting source /data/research/vcpkg/downloads/wolfpld-tracy-v0.10.tar.gz
-- Applying patch 001-fix-vcxproj-vcpkg.patch
-- Applying patch 002-use-internal-imgui.patch
-- Using source at /data/research/vcpkg/buildtrees/tracy/src/v0.10-b0e1fd57f2.clean
-- Configuring x64-linux
-- Installing: /data/research/vcpkg/packages/tracy_x64-linux/share/tracy/copyright
[DEBUG] 1011: cmd_execute_and_stream_data() returned 0 after   153689 us
-- Performing post-build validation

可以看到，其关键是用内置的CMake调用了vcpkg的ports.cmake脚本，并传入了相关参数，如：

ALL_FEATURES / TARGET_TRIPLET / TARGET_TRIPLET_FILE
CURRENT_PORT_DIR
Z_VCPKG_CHAINLOAD_TOOLCHAIN_FILE
….

至此，我们可以发现即使是自定义的portfile.cmake，其环境也和主CMake工程是隔离的，无法直接传入全局CMake变量。

Step3: porfile.cmake阶段

接下来，分析portfile的逻辑，脚本不长，主要是两部分：

# 清理变量，并include各种脚本依赖
.....
set(SCRIPTS "${CMAKE_CURRENT_LIST_DIR}" CACHE PATH "Location to stored scripts")
list(APPEND CMAKE_MODULE_PATH "${SCRIPTS}/cmake")
include("${SCRIPTS}/cmake/execute_process.cmake")
include("${SCRIPTS}/cmake/vcpkg_acquire_msys.cmake")
include("${SCRIPTS}/cmake/vcpkg_add_to_path.cmake")
include("${SCRIPTS}/cmake/vcpkg_apply_patches.cmake")
include("${SCRIPTS}/cmake/vcpkg_backup_restore_env_vars.cmake")
include("${SCRIPTS}/cmake/vcpkg_build_cmake.cmake")
include("${SCRIPTS}/cmake/vcpkg_build_make.cmake")
.....
# 进行port的构建
if(CMD STREQUAL "BUILD")
    # VNote: 检查必要的参数，省略
    include("${CURRENT_PORT_DIR}/portfile.cmake") # VNote: 触发port的构建
    if(DEFINED PORT)
        if(VCPKG_FIXUP_ELF_RPATH)
            include("${SCRIPTS}/cmake/z_vcpkg_fixup_rpath.cmake")
        endif()
        include("${SCRIPTS}/build_info.cmake")
    endif()
....

接下来，就是我们最熟悉的portfile.cmake的执行了，以tracy为例，其portfile.cmake如下(注：相关的api已经在上面的ports.cmake中include):

vcpkg_check_linkage(ONLY_STATIC_LIBRARY)
# VNote: 从github下载vcpkg的源码
vcpkg_from_github(  ..... )
vcpkg_check_features(OUT_FEATURE_OPTIONS FEATURE_OPTIONS .... )

# VNote: tracy本身是cmake构建的，则进行配置
vcpkg_cmake_configure(
    SOURCE_PATH ${SOURCE_PATH}
    OPTIONS ${FEATURE_OPTIONS}
)
# VNote: 构建并安装tracy
vcpkg_cmake_install()

# VNote: 省略一些特定平台的处理，下面是Windows平台的构建
function(tracy_tool_install tracy_TOOL tracy_TOOL_NAME)
    if(VCPKG_TARGET_IS_WINDOWS)
        tracy_tool_install_win32("${tracy_TOOL}" "${tracy_TOOL_NAME}")
    else()
        tracy_tool_install_unix("${tracy_TOOL}" "${tracy_TOOL_NAME}")
    endif()
endfunction()

#VNote: 省略....
vcpkg_install_copyright(FILE_LIST "${SOURCE_PATH}/LICENSE")
file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/share")
file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/debug/include")

以上逻辑中，最关键的是:

vcpkg_cmake_configure: 位于ports/vcpkg-cmake/vcpkg_cmake_configure.cmake，组装cmake命令
vcpkg_cmake_install: 位于ports/vcpkg-cmake/vcpkg_cmake_install.cmake，调用vcpkg_cmake_build
vcpkg_cmake_build: 位于ports/vcpkg-cmake/vcpkg_cmake_build.cmake，进行最终的构建和cmake install(项目的CMake脚本实现)

以vcpkg install tarcy:arm-neon-android为例(linux的vcpkg install tarcy过于简单，选取个复杂的)，其vcpkg_cmake_configure最终组装的cmake命令如下(可以想象，如果要我们手动构建并正确配置参数，还是比较复杂的)：

# 为了便于理解，进行换行处理
[1/2] "/path/to/.vcpkg/downloads/tools/cmake-3.29.2-linux/cmake-3.29.2-linux-x86_64/bin/cmake" \
  -E chdir ".." "/path/to/.vcpkg/downloads/tools/cmake-3.29.2-linux/cmake-3.29.2-linux-x86_64/bin/cmake" \
  "/path/to/.vcpkg/buildtrees/tracy/src/v0.10-b0e1fd57f2.clean" \
  "-G" "Ninja" "-DCMAKE_BUILD_TYPE=Release" \
  "-DCMAKE_INSTALL_PREFIX=/path/to/.vcpkg/packages/tracy_arm-neon-android" \
  "-DFETCHCONTENT_FULLY_DISCONNECTED=ON" \
  "-DTRACY_ON_DEMAND=OFF" "-DTRACY_FIBERS=OFF" "-DTRACY_NO_CRASH_HANDLER=OFF" \
  "-DCMAKE_MAKE_PROGRAM=/path/to/.vcpkg/downloads/tools/ninja/1.10.2-linux/ninja" \
  "-DCMAKE_SYSTEM_NAME=Android" "-DCMAKE_SYSTEM_VERSION=21" "-DBUILD_SHARED_LIBS=OFF" \
  "-DVCPKG_CHAINLOAD_TOOLCHAIN_FILE=/path/to/.vcpkg/scripts/toolchains/android.cmake" \
  "-DVCPKG_TARGET_TRIPLET=arm-neon-android" "-DVCPKG_SET_CHARSET_FLAG=ON" \
  "-DVCPKG_PLATFORM_TOOLSET=external" "-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON" \
  "-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON" "-DCMAKE_FIND_PACKAGE_NO_SYSTEM_PACKAGE_REGISTRY=ON" \
  "-DCMAKE_INSTALL_SYSTEM_RUNTIME_LIBS_SKIP=TRUE" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-DVCPKG_APPLOCAL_DEPS=OFF" \
  "-DCMAKE_TOOLCHAIN_FILE=/path/to/.vcpkg/scripts/buildsystems/vcpkg.cmake" \
  "-DCMAKE_ERROR_ON_ABSOLUTE_INSTALL_DESTINATION=ON" \
  "-DVCPKG_CXX_FLAGS=" "-DVCPKG_CXX_FLAGS_RELEASE=" "-DVCPKG_CXX_FLAGS_DEBUG=" \
  "-DVCPKG_C_FLAGS=" "-DVCPKG_C_FLAGS_RELEASE=" "-DVCPKG_C_FLAGS_DEBUG=" \
  "-DVCPKG_CRT_LINKAGE=static" "-DVCPKG_LINKER_FLAGS=" "-DVCPKG_LINKER_FLAGS_RELEASE=" "-DVCPKG_LINKER_FLAGS_DEBUG=" \
  "-DVCPKG_TARGET_ARCHITECTURE=arm" "-DCMAKE_INSTALL_LIBDIR:STRING=lib" "-DCMAKE_INSTALL_BINDIR:STRING=bin" \
  "-D_VCPKG_ROOT_DIR=/path/to/.vcpkg" "-D_VCPKG_INSTALLED_DIR=/path/to/android-build/Release/vcpkg_installed" \
  "-DVCPKG_MANIFEST_INSTALL=OFF" "-DANDROID_ABI=armeabi-v7a" "-DANDROID_ARM_NEON=ON"

其vcpkg_cmake_build将进行最终构建:

# 为了便于理解，进行换行处理
Run Build Command(s): /path/to/.vcpkg/downloads/tools/ninja/1.10.2-linux/ninja -v -v -j17 install
[1/3] /root/.install/android-sdk-linux/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ \
  --target=armv7-none-linux-androideabi21 \
  --sysroot=/root/.install/android-sdk-linux/ndk/26.1.10909125/toolchains/llvm/prebuilt/linux-x86_64/sysroot \
  -DTRACY_ENABLE -isystem /path/to/.vcpkg/buildtrees/tracy/src/v0.10-b0e1fd57f2.clean/public \
  -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 \
  -march=armv7-a -mthumb -Wformat -Werror=format-security -frtti -fexceptions  -fPIC   -fno-limit-debug-info    -fPIC -pthread -MD -MT \
  CMakeFiles/TracyClient.dir/public/TracyClient.cpp.o \
  -MF CMakeFiles/TracyClient.dir/public/TracyClient.cpp.o.d \
  -o CMakeFiles/TracyClient.dir/public/TracyClient.cpp.o \
  -c /path/to/.vcpkg/buildtrees/tracy/src/v0.10-b0e1fd57f2.clean/public/TracyClient.cpp

总结

至此，我们理清了主CMake工程与其依赖的port（三方库）的portfile.cmake脚本的关系，有时其虽然在同一个项目中，但其实彼此是完全隔离的，这个不看源码，是很容易误解的。(cmake -> vcpkg-tool -> portfile.cmake的转发，对调试是一种挑战，所以porfile.cmake的问题尽可能通过日志排查)
另外，也可以直接感受到，vcpkg把一个三方库的各种构建细节（依赖、交叉编译、参数选择等）进行了封装，让使用者尽可能不关注依赖的构建细节（如果这个依赖的构建脚本维护良好的话）。

总阅读量次。