Skip to content

Use vcpkgs for dependencies on Windows #3490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Magnus Edenhill (edenhill) merged 2 commits into from
Aug 12, 2021
Merged

Use vcpkgs for dependencies on Windows #3490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
902c5be
Appveyor: build with VS 2015 and msvcr 140, use vcpkg for deps
edenhill Mar 29, 2021
75c4314
Use vcpkg version pinning(ish)
edenhill Aug 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 24 additions & 14 deletions .appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,33 @@
version: 1.0.0-R-post{build}
version: 1.8.0-R-post{build}
pull_requests:
do_not_increment_build_number: true
image: Visual Studio 2013
image: Visual Studio 2015
configuration: Release
environment:
runtime: v140
matrix:
- platform: x64
arch: x64
- platform: win32
arch: x86
install:
- ps: "& .\\win32\\install-openssl.ps1"
- ps: "& .\\win32\\install-coapp.ps1"
# Update vcpkg (is outdated on the VS 2015 image)
- cmd: |
cd "C:\Tools\vcpkg"
git pull -q
.\bootstrap-vcpkg.bat
cd %appveyor_build_folder%
cache:
- c:\OpenSSL-Win32 -> win32\install-openssl.ps1
- c:\OpenSSL-Win64 -> win32\install-openssl.ps1
- c:\tools\vcpkg\installed
- C:\Users\appveyor\AppData\Local\vcpkg\archives
- C:\Users\appveyor\AppData\Local\vcpkg\installed
nuget:
account_feed: true
project_feed: true
disable_publish_on_pr: true
before_build:
- cmd: nuget restore win32/librdkafka.sln
- cmd: vcpkg --feature-flags=versions install --triplet %arch%-windows
build:
project: win32/librdkafka.sln
publish_nuget: true
Expand All @@ -27,7 +36,7 @@ build:
parallel: true
verbosity: normal
test_script:
- cmd: cd tests && ..\win32\outdir\v120\%PLATFORM%\%CONFIGURATION%\tests.exe -l -Q -p1 && cd ..
- cmd: cd tests && ..\win32\outdir\%runtime%\%PLATFORM%\%CONFIGURATION%\tests.exe -l -Q -p1 && cd ..
artifacts:
- path: test_report*.json
name: Test report
Expand All @@ -41,7 +50,8 @@ artifacts:
name: Libraries
- path: '**\*.exe'
name: Executables
before_deploy:
#before_deploy:
after_test:
- ps: >-
# FIXME: Add to Deployment condition above:

Expand All @@ -53,13 +63,13 @@ before_deploy:

$autopkgFile = "win32/librdkafka.autopkg"

pwd
pwd


ls $autopkgFile


ls $autopkgFile



# Get the ".autopkg.template" file, replace "@version" with the Appveyor version number, then save to the ".autopkg" file.

cat ($autopkgFile + ".template") | % { $_ -replace "@version", $env:appveyor_build_version } > $autopkgFile
Expand All @@ -76,9 +86,9 @@ before_deploy:
deploy:
- provider: S3
access_key_id:
secure: iBK0xb23FMYOrOsOb8cw3YGyU+6vvPX5BF+PXuMub8M=
secure: iBK0xb23FMYOrOsOb8cw3YGyU+6vvPX5BF+PXuMub8M=
secret_access_key:
secure: jJsj373UiOtuXf/u0LLL0Q8XQMyu4s/ucx0+vH4GpKbAfZJUwYB4dEO1//mQDNuC
secure: jJsj373UiOtuXf/u0LLL0Q8XQMyu4s/ucx0+vH4GpKbAfZJUwYB4dEO1//mQDNuC
region: us-west-1
bucket: librdkafka-ci-packages
folder: librdkafka/p-librdkafka__bld-appveyor__plat-windows__arch-$(platform)__bldtype-$(configuration)__tag-$(APPVEYOR_REPO_TAG_NAME)__sha-$(APPVEYOR_REPO_COMMIT)__bid-$(APPVEYOR_BUILD_ID)
Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,19 @@ librdkafka v1.7.0 is feature release:
* The binary librdkafka artifacts for Alpine are now using Alpine 3.12.
OpenSSL 1.1.1k.
* Improved static librdkafka Windows builds using MinGW (@neptoess, #3130).
* The `librdkafka.redist` NuGet package now has updated zlib, zstd and
OpenSSL versions (from vcpkg).


## Security considerations

* The zlib version bundled with the `librdkafka.redist` NuGet package has now been upgraded
from zlib 1.2.8 to 1.2.11, fixing the following CVEs:
* CVE-2016-9840: undefined behaviour (compiler dependent) in inflate (decompression) code: this is used by the librdkafka consumer. Risk of successfully exploitation through consumed messages is eastimated very low.
* CVE-2016-9841: undefined behaviour (compiler dependent) in inflate code: this is used by the librdkafka consumer. Risk of successfully exploitation through consumed messages is eastimated very low.
* CVE-2016-9842: undefined behaviour in inflateMark(): this API is not used by librdkafka.
* CVE-2016-9843: issue in crc32_big() which is called from crc32_z(): this API is not used by librdkafka.

## Upgrade considerations

* The C++ `oauthbearer_token_refresh_cb()` was missing a `Handle *`
Expand All @@ -79,6 +90,8 @@ librdkafka v1.7.0 is feature release:
This is more correct than the previous `consumer_lag` which was using
either `committed_offset` or `app_offset` (last message passed
to application).
* The `librdkafka.redist` NuGet package is now built with MSVC runtime v140
(VS 2015). Previous versions were built with MSVC runtime v120 (VS 2013).


## Fixes
Expand Down
3 changes: 2 additions & 1 deletion packaging/RELEASE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,10 @@ Release candidates start at 200, thus 0xAABBCCc9 is RC1, 0xAABBCCca is RC2, etc.
Change the `RD_KAFKA_VERSION` defines in both `src/rdkafka.h` and
`src-cpp/rdkafkacpp.h` to the version to build, such as 0x000b01c9
for v0.11.1-RC1, or 0x000b01ff for the final v0.11.1 release.
Update the librdkafka version in `vcpkg.json`.

# Update defines
$ $EDITOR src/rdkafka.h src-cpp/rdkafkacpp.h
$ $EDITOR src/rdkafka.h src-cpp/rdkafkacpp.h vcpkg.json

# Reconfigure and build
$ ./configure
Expand Down
125 changes: 68 additions & 57 deletions packaging/nuget/packaging.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,19 +327,19 @@ def build (self, buildtype):
a.info.get('arch'),
a.info.get('bldtype'))
if 'toolset' not in a.info:
a.info['toolset'] = 'v120'
Copy link
Contributor

@mhowlett Matt Howlett (mhowlett) Aug 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if this affects OOTB compatibility anywhere important, but 2015 was a long time ago...

Copy link
Contributor Author

@edenhill Magnus Edenhill (edenhill) Aug 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm not completely sure either. But we'll find out

a.info['toolset'] = 'v140'

mappings = [
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka.h', 'build/native/include/librdkafka/rdkafka.h'],
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafkacpp.h', 'build/native/include/librdkafka/rdkafkacpp.h'],
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka_mock.h', 'build/native/include/librdkafka/rdkafka_mock.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka.h', 'build/native/include/librdkafka/rdkafka.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafkacpp.h', 'build/native/include/librdkafka/rdkafkacpp.h'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './include/librdkafka/rdkafka_mock.h', 'build/native/include/librdkafka/rdkafka_mock.h'],

# Travis OSX build
[{'arch': 'x64', 'plat': 'osx', 'fname_glob': 'librdkafka-clang.tar.gz'}, './lib/librdkafka.dylib', 'runtimes/osx-x64/native/librdkafka.dylib'],
# Travis Manylinux build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-manylinux*x86_64.tgz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/centos6-librdkafka.so'],
# Travis Ubuntu 14.04 build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/librdkafka.so'],
[{'arch': 'x64', 'plat': 'linux', 'lnk': 'std', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-x64/native/librdkafka.so'],
# Travis CentOS 7 RPM build
[{'arch': 'x64', 'plat': 'linux', 'fname_glob': 'librdkafka1*el7.x86_64.rpm'}, './usr/lib64/librdkafka.so.1', 'runtimes/linux-x64/native/centos7-librdkafka.so'],
# Travis Alpine build
Expand All @@ -348,32 +348,37 @@ def build (self, buildtype):
[{'arch': 'arm64', 'plat': 'linux', 'fname_glob': 'librdkafka-gcc.tar.gz'}, './lib/librdkafka.so.1', 'runtimes/linux-arm64/native/librdkafka.so'],

# Common Win runtime
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcr120.dll', 'runtimes/win-x64/native/msvcr120.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcp120.dll', 'runtimes/win-x64/native/msvcp120.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'vcruntime140.dll', 'runtimes/win-x64/native/vcruntime140.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'msvcp140.dll', 'runtimes/win-x64/native/msvcp140.dll'],
# matches librdkafka.redist.{VER}.nupkg
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/librdkafka.dll', 'runtimes/win-x64/native/librdkafka.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/librdkafkacpp.dll', 'runtimes/win-x64/native/librdkafkacpp.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/zlib.dll', 'runtimes/win-x64/native/zlib.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/x64/Release/libzstd.dll', 'runtimes/win-x64/native/libzstd.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/librdkafka.dll', 'runtimes/win-x64/native/librdkafka.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/librdkafkacpp.dll', 'runtimes/win-x64/native/librdkafkacpp.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/libcrypto-1_1-x64.dll', 'runtimes/win-x64/native/libcrypto-1_1-x64.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/libssl-1_1-x64.dll', 'runtimes/win-x64/native/libssl-1_1-x64.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/zlib1.dll', 'runtimes/win-x64/native/zlib1.dll'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/x64/Release/zstd.dll', 'runtimes/win-x64/native/zstd.dll'],
# matches librdkafka.{VER}.nupkg
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/x64/Release/librdkafka.lib', 'build/native/lib/win/x64/win-x64-Release/v120/librdkafka.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/x64/Release/librdkafkacpp.lib', 'build/native/lib/win/x64/win-x64-Release/v120/librdkafkacpp.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/x64/Release/librdkafka.lib', 'build/native/lib/win/x64/win-x64-Release/v140/librdkafka.lib'],
[{'arch': 'x64', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/x64/Release/librdkafkacpp.lib', 'build/native/lib/win/x64/win-x64-Release/v140/librdkafkacpp.lib'],

[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcr120.dll', 'runtimes/win-x86/native/msvcr120.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr120.zip'}, 'msvcp120.dll', 'runtimes/win-x86/native/msvcp120.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'vcruntime140.dll', 'runtimes/win-x86/native/vcruntime140.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'msvcr140.zip'}, 'msvcp140.dll', 'runtimes/win-x86/native/msvcp140.dll'],
# matches librdkafka.redist.{VER}.nupkg
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/librdkafka.dll', 'runtimes/win-x86/native/librdkafka.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/librdkafkacpp.dll', 'runtimes/win-x86/native/librdkafkacpp.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/zlib.dll', 'runtimes/win-x86/native/zlib.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v120/Win32/Release/libzstd.dll', 'runtimes/win-x86/native/libzstd.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/librdkafka.dll', 'runtimes/win-x86/native/librdkafka.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/librdkafkacpp.dll', 'runtimes/win-x86/native/librdkafkacpp.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/libcrypto-1_1.dll', 'runtimes/win-x86/native/libcrypto-1_1.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/libssl-1_1.dll', 'runtimes/win-x86/native/libssl-1_1.dll'],

[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/zlib1.dll', 'runtimes/win-x86/native/zlib1.dll'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka.redist*'}, 'build/native/bin/v140/Win32/Release/zstd.dll', 'runtimes/win-x86/native/zstd.dll'],

# matches librdkafka.{VER}.nupkg
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/Win32/Release/librdkafka.lib', 'build/native/lib/win/x86/win-x86-Release/v120/librdkafka.lib'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v120/Win32/Release/librdkafkacpp.lib', 'build/native/lib/win/x86/win-x86-Release/v120/librdkafkacpp.lib']
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/Win32/Release/librdkafka.lib', 'build/native/lib/win/x86/win-x86-Release/v140/librdkafka.lib'],
[{'arch': 'x86', 'plat': 'win', 'fname_glob': 'librdkafka*.nupkg', 'fname_excludes': ['redist', 'symbols']},
'build/native/lib/v140/Win32/Release/librdkafkacpp.lib', 'build/native/lib/win/x86/win-x86-Release/v140/librdkafkacpp.lib']
]

for m in mappings:
Expand All @@ -385,36 +390,36 @@ def build (self, buildtype):
fname_excludes = attributes['fname_excludes']
del attributes['fname_excludes']

artifact = None
for a in self.arts.artifacts:
found = True
outf = os.path.join(self.stpath, m[2])
member = m[1]

found = False
# Try all matching artifacts until we find the wanted file (member)
for a in self.arts.artifacts:
for attr in attributes:
if a.info[attr] != attributes[attr]:
found = False
break
if a.info.get(attr, None) != attributes[attr]:
continue

if not fnmatch(a.fname, fname_glob):
found = False
continue

for exclude in fname_excludes:
if exclude in a.fname:
found = False
break
continue

if found:
artifact = a
break
try:
zfile.ZFile.extract(a.lpath, member, outf)
except KeyError as e:
continue
except Exception as e:
raise Exception('file not found in archive %s: %s. Files in archive are: %s' % (a.lpath, e, zfile.ZFile(a.lpath).getnames()))

if artifact is None:
raise MissingArtifactError('unable to find artifact with tags %s matching "%s"' % (str(attributes), fname_glob))
found = True
break

if not found:
raise MissingArtifactError('unable to find artifact with tags %s matching "%s" for file "%s"' % (str(attributes), fname_glob, member))

outf = os.path.join(self.stpath, m[2])
member = m[1]
try:
zfile.ZFile.extract(artifact.lpath, member, outf)
except KeyError as e:
raise Exception('file not found in archive %s: %s. Files in archive are: %s' % (artifact.lpath, e, zfile.ZFile(artifact.lpath).getnames()))

print('Tree extracted to %s' % self.stpath)

Expand All @@ -437,28 +442,34 @@ def verify (self, path):
"build/native/include/librdkafka/rdkafka.h",
"build/native/include/librdkafka/rdkafkacpp.h",
"build/native/include/librdkafka/rdkafka_mock.h",
"build/native/lib/win/x64/win-x64-Release/v120/librdkafka.lib",
"build/native/lib/win/x64/win-x64-Release/v120/librdkafkacpp.lib",
"build/native/lib/win/x86/win-x86-Release/v120/librdkafka.lib",
"build/native/lib/win/x86/win-x86-Release/v120/librdkafkacpp.lib",
"build/native/lib/win/x64/win-x64-Release/v140/librdkafka.lib",
"build/native/lib/win/x64/win-x64-Release/v140/librdkafkacpp.lib",
"build/native/lib/win/x86/win-x86-Release/v140/librdkafka.lib",
"build/native/lib/win/x86/win-x86-Release/v140/librdkafkacpp.lib",
"runtimes/linux-x64/native/centos7-librdkafka.so",
"runtimes/linux-x64/native/centos6-librdkafka.so",
"runtimes/linux-x64/native/alpine-librdkafka.so",
"runtimes/linux-x64/native/librdkafka.so",
"runtimes/linux-arm64/native/librdkafka.so",
"runtimes/osx-x64/native/librdkafka.dylib",
# win x64
"runtimes/win-x64/native/librdkafka.dll",
"runtimes/win-x64/native/librdkafkacpp.dll",
"runtimes/win-x64/native/msvcr120.dll",
"runtimes/win-x64/native/msvcp120.dll",
"runtimes/win-x64/native/zlib.dll",
"runtimes/win-x64/native/libzstd.dll",
"runtimes/win-x64/native/vcruntime140.dll",
"runtimes/win-x64/native/msvcp140.dll",
"runtimes/win-x64/native/libcrypto-1_1-x64.dll",
"runtimes/win-x64/native/libssl-1_1-x64.dll",
"runtimes/win-x64/native/zlib1.dll",
"runtimes/win-x64/native/zstd.dll",
# win x86
"runtimes/win-x86/native/librdkafka.dll",
"runtimes/win-x86/native/librdkafkacpp.dll",
"runtimes/win-x86/native/msvcr120.dll",
"runtimes/win-x86/native/msvcp120.dll",
"runtimes/win-x86/native/zlib.dll",
"runtimes/win-x86/native/libzstd.dll"]
"runtimes/win-x86/native/vcruntime140.dll",
"runtimes/win-x86/native/msvcp140.dll",
"runtimes/win-x86/native/libcrypto-1_1.dll",
"runtimes/win-x86/native/libssl-1_1.dll",
"runtimes/win-x86/native/zlib1.dll",
"runtimes/win-x86/native/zstd.dll"]

missing = list()
with zfile.ZFile(path, 'r') as zf:
Expand Down
Loading