nprasad2077/doris
1
0 Fork 0
mirror of https://github.com/apache/doris.git synced 2026-11-04 07:31:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
378404eeb12a1d646a5a5e725cf7f79e5170960e
doris/thirdparty/vars.sh
T
Gabriel 378404eeb1 [fix](thirdparty) Upgrade Brotli to 1.0.10 for CVE-2026-111497 (#69906) 
### What problem does this PR solve?

Issue Number: None

Related PR: None

Doris bundles Brotli 1.0.9, which is affected by
[CVE-2026-111497](https://github.com/google/brotli/releases/tag/v1.0.10).
The upstream decoder can mistake an invalid block switch for
insufficient input, allowing an out-of-bounds write.

Upgrade the bundled dependency to the upstream 1.0.10 security release
and update its archive checksum. Align Arrow 24's internal Brotli
version and SHA256 pin with the overridden archive; changing the Doris
manifest alone causes Arrow's ExternalProject verification to fail.
Apply that metadata patch with a separate marker so previously patched
Arrow source trees also receive it.

The fix requires rebuilding the relevant thirdparty libraries and
relinking BE because the decoder is statically linked.

### Release note

Upgrade bundled Brotli to 1.0.10 to include the upstream fix for
CVE-2026-111497.

### Check List (For Author)

- Test: Manual validation passed:
- Official source archive MD5 and SHA256 verification; the source
contains the upstream block-switch error fix.
  - Bash syntax and `git diff --check`.
  - Existing thirdparty download checksum and fallback tests.
- Actual Brotli/Arrow companion download and extraction; patch
application to previously patched Arrow sources and repeated invocation.
  - Patch dry-run against pristine Arrow 24 metadata.
- Arrow 24's Brotli ExternalProject verification, build, and
static-library installation in an isolated CMake harness. Confirmed the
old SHA256 pin fails with the new archive before the patch, then passes
after it.
- Brotli 1.0.10 build and all 14 available upstream
round-trip/compatibility tests with AddressSanitizer and
UndefinedBehaviorSanitizer.
- Full Doris and Arrow builds were not run locally; CI validation is
requested.
- Behavior changed: Yes. Builds use the patched Brotli decoder and
updated Arrow dependency metadata.
- Does this need documentation: No. This updates a bundled security
dependency.
2026-11-02 10:10:45 +08:00

747 lines
28 KiB
Bash
Raw Blame History

#!/bin/bash
# shellcheck disable=2034
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
############################################################
# You may have to set variables bellow,
# which are used for compiling thirdparties and palo itself.
############################################################
###################################################
# DO NOT change variables bellow unless you known
# what you are doing.
###################################################
# thirdparties will be downloaded and unpacked here
export TP_SOURCE_DIR="${TP_DIR:-.}/src"
# thirdparties will be installed to here
export TP_INSTALL_DIR="${TP_DIR:-.}/installed"
# patches for all thirdparties
export TP_PATCH_DIR="${TP_DIR:-.}/patches"
# header files of all thirdparties will be intalled to here
export TP_INCLUDE_DIR="${TP_INSTALL_DIR}/include"
# libraries of all thirdparties will be intalled to here
export TP_LIB_DIR="${TP_INSTALL_DIR}/lib"
# all java libraries will be unpacked to here
export TP_JAR_DIR="${TP_INSTALL_DIR}/lib/jar"
# source of all dependencies, default unuse it
# export REPOSITORY_URL=
DORIS_THIRDPARTY_REPOSITORY_URL="${DORIS_THIRDPARTY_REPOSITORY_URL:-https://doris-regression-hk.oss-cn-hongkong.aliyuncs.com/regression/datalake/thirdparty/juicefs}"
#####################################################
# Download url, filename and unpaced filename
# of all thirdparties
#####################################################
# libevent
LIBEVENT_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/libevent-2.1.12.1.tar.gz"
LIBEVENT_NAME=libevent-2.1.12.1.tar.gz
LIBEVENT_SOURCE=doris-thirdparty-libevent-2.1.12.1
LIBEVENT_MD5SUM="af67af024d42e57e185eb0c5026f8ae6"
# openssl
OPENSSL_DOWNLOAD="https://github.com/openssl/openssl/archive/OpenSSL_1_1_1s.tar.gz"
OPENSSL_NAME=openssl-OpenSSL_1_1_1s.tar.gz
OPENSSL_SOURCE=openssl-OpenSSL_1_1_1s
OPENSSL_MD5SUM="7e79a7560dee77c0758baa33c61af4b4"
# thrift
THRIFT_DOWNLOAD="http://archive.apache.org/dist/thrift/0.24.0/thrift-0.24.0.tar.gz"
THRIFT_NAME=thrift-0.24.0.tar.gz
THRIFT_SOURCE=thrift-0.24.0
THRIFT_MD5SUM="232e035ff80c5fb4b7243f0be3a76b02"
# protobuf
# brpc is not yet compatible with protobuf >= 22
PROTOBUF_DOWNLOAD="https://github.com/protocolbuffers/protobuf/releases/download/v21.11/protobuf-all-21.11.tar.gz"
PROTOBUF_NAME="protobuf-all-21.11.tar.gz"
PROTOBUF_SOURCE=protobuf-21.11
PROTOBUF_MD5SUM="b3b104f0374802e1add5d5d7a5a845ac"
# gflags
GFLAGS_DOWNLOAD="https://github.com/gflags/gflags/archive/v2.2.2.tar.gz"
GFLAGS_NAME=gflags-2.2.2.tar.gz
GFLAGS_SOURCE=gflags-2.2.2
GFLAGS_MD5SUM="1a865b93bacfa963201af3f75b7bd64c"
# glog
GLOG_DOWNLOAD="https://github.com/google/glog/archive/refs/tags/v0.6.0.tar.gz"
GLOG_NAME="glog-v0.6.0.tar.gz"
GLOG_SOURCE=glog-0.6.0
GLOG_MD5SUM="c98a6068bc9b8ad9cebaca625ca73aa2"
# gtest
GTEST_DOWNLOAD="https://github.com/google/googletest/archive/refs/tags/release-1.12.1.tar.gz"
GTEST_NAME=googletest-release-1.12.1.tar.gz
GTEST_SOURCE=googletest-release-1.12.1
GTEST_MD5SUM="e82199374acdfda3f425331028eb4e2a"
# snappy
SNAPPY_DOWNLOAD="https://github.com/google/snappy/archive/1.2.1.tar.gz"
SNAPPY_NAME=snappy-1.2.1.tar.gz
SNAPPY_SOURCE=snappy-1.2.1
SNAPPY_MD5SUM="dd6f9b667e69491e1dbf7419bdf68823"
# gperftools
GPERFTOOLS_DOWNLOAD="https://github.com/gperftools/gperftools/releases/download/gperftools-2.18.1/gperftools-2.18.1.tar.gz"
GPERFTOOLS_NAME=gperftools-2.18.1.tar.gz
GPERFTOOLS_SOURCE=gperftools-2.18.1
GPERFTOOLS_MD5SUM="129c01f6f5297f0482b33d431b5ec555"
# zlib
ZLIB_DOWNLOAD="https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz"
ZLIB_NAME=zlib-1.3.1.tar.gz
ZLIB_SOURCE=zlib-1.3.1
ZLIB_MD5SUM="9855b6d802d7fe5b7bd5b196a2271655"
# crc32c
CRC32C_DOWNLOAD="https://github.com/google/crc32c/archive/refs/tags/1.1.2.tar.gz"
CRC32C_NAME=crc32c-1.1.2.tar.gz
CRC32C_SOURCE=crc32c-1.1.2
CRC32C_MD5SUM="cc0338e6a60c38cab04a70a2c36cd9f2"
# lz4
LZ4_DOWNLOAD="https://github.com/lz4/lz4/archive/v1.9.4.tar.gz"
LZ4_NAME=lz4-1.9.4.tar.gz
LZ4_SOURCE=lz4-1.9.4
LZ4_MD5SUM="e9286adb64040071c5e23498bf753261"
# bzip
BZIP_DOWNLOAD="https://fossies.org/linux/misc/bzip2-1.0.8.tar.gz"
BZIP_NAME=bzip2-1.0.8.tar.gz
BZIP_SOURCE=bzip2-1.0.8
BZIP_MD5SUM="67e051268d0c475ea773822f7500d0e5"
# lzo2
LZO2_DOWNLOAD="https://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz"
LZO2_NAME=lzo-2.10.tar.gz
LZO2_SOURCE=lzo-2.10
LZO2_MD5SUM="39d3f3f9c55c87b1e5d6888e1420f4b5"
# rapidjson
RAPIDJSON_DOWNLOAD="https://github.com/miloyip/rapidjson/archive/232389d4f1012dddec4ef84861face2d2ba85709.tar.gz"
RAPIDJSON_NAME=rapidjson-232389d4f1012dddec4ef84861face2d2ba85709.tar.gz
RAPIDJSON_SOURCE=rapidjson-232389d4f1012dddec4ef84861face2d2ba85709
RAPIDJSON_MD5SUM="577d3495a07b66fcd4a2866c93831bc4"
# curl
CURL_DOWNLOAD="https://curl.se/download/curl-8.2.1.tar.gz"
CURL_NAME="curl-8.2.1.tar.gz"
CURL_SOURCE=curl-8.2.1
CURL_MD5SUM="b25588a43556068be05e1624e0e74d41"
# RE2
RE2_DOWNLOAD="https://github.com/google/re2/archive/2021-02-02.tar.gz"
RE2_NAME=re2-2021-02-02.tar.gz
RE2_SOURCE=re2-2021-02-02
RE2_MD5SUM="48bc665463a86f68243c5af1bac75cd0"
# hyperscan
HYPERSCAN_DOWNLOAD="https://github.com/intel/hyperscan/archive/refs/tags/v5.4.2.tar.gz"
HYPERSCAN_NAME=hyperscan-5.4.2.tar.gz
HYPERSCAN_SOURCE=hyperscan-5.4.2
HYPERSCAN_MD5SUM="202f4b42f5dd4a7bb2506445e51a33b9"
# vectorscan (support arm for hyperscan)
MACHINE_TYPE=$(uname -m)
if [[ "${MACHINE_TYPE}" == "aarch64" || "${MACHINE_TYPE}" == 'arm64' ]]; then
echo "use vectorscan instead of hyperscan on aarch64"
HYPERSCAN_DOWNLOAD="https://github.com/VectorCamp/vectorscan/archive/refs/tags/vectorscan/5.4.11.tar.gz"
HYPERSCAN_NAME=vectorscan-5.4.11.tar.gz
HYPERSCAN_SOURCE=vectorscan-vectorscan-5.4.11
HYPERSCAN_MD5SUM="e67b70403cba6c1654a9fef4fd15a2f2"
fi
# ragel (dependency for hyperscan)
RAGEL_DOWNLOAD="http://www.colm.net/files/ragel/ragel-6.10.tar.gz"
RAGEL_NAME=ragel-6.10.tar.gz
RAGEL_SOURCE=ragel-6.10
RAGEL_MD5SUM="748cae8b50cffe9efcaa5acebc6abf0d"
# boost
BOOST_DOWNLOAD="https://archives.boost.io/release/1.81.0/source/boost_1_81_0.tar.gz"
BOOST_NAME=boost_1_81_0.tar.gz
BOOST_SOURCE=boost_1_81_0
BOOST_MD5SUM="4bf02e84afb56dfdccd1e6aec9911f4b"
# mysql
MYSQL_DOWNLOAD="https://github.com/mysql/mysql-server/archive/mysql-5.7.18.tar.gz"
MYSQL_NAME=mysql-5.7.18.tar.gz
MYSQL_SOURCE=mysql-server-mysql-5.7.18
MYSQL_MD5SUM="58598b10dce180e4d1fbdd7cf5fa68d6"
# unix odbc
ODBC_DOWNLOAD="http://www.unixodbc.org/unixODBC-2.3.7.tar.gz"
ODBC_NAME=unixODBC-2.3.7.tar.gz
ODBC_SOURCE=unixODBC-2.3.7
ODBC_MD5SUM="274a711b0c77394e052db6493840c6f9"
# leveldb
LEVELDB_DOWNLOAD="https://github.com/google/leveldb/archive/refs/tags/1.23.tar.gz"
LEVELDB_NAME=leveldb-1.23.tar.gz
LEVELDB_SOURCE=leveldb-1.23
LEVELDB_MD5SUM="afbde776fb8760312009963f09a586c7"
# brpc
BRPC_DOWNLOAD="https://github.com/apache/brpc/archive/refs/tags/1.4.0.tar.gz"
BRPC_NAME="brpc-1.4.0.tar.gz"
BRPC_SOURCE="brpc-1.4.0"
BRPC_MD5SUM="6af9d50822c33a3abc56a1ec0af0e0bc"
# rocksdb
ROCKSDB_DOWNLOAD="https://github.com/facebook/rocksdb/archive/v5.14.2.tar.gz"
ROCKSDB_NAME=rocksdb-5.14.2.tar.gz
ROCKSDB_SOURCE=rocksdb-5.14.2
ROCKSDB_MD5SUM="b72720ea3b1e9ca9e4ed0febfef65b14"
# cyrus-sasl
CYRUS_SASL_DOWNLOAD="https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.27/cyrus-sasl-2.1.27.tar.gz"
CYRUS_SASL_NAME=cyrus-sasl-2.1.27.tar.gz
CYRUS_SASL_SOURCE=cyrus-sasl-2.1.27
CYRUS_SASL_MD5SUM="a33820c66e0622222c5aefafa1581083"
# librdkafka-2.16.0
LIBRDKAFKA_DOWNLOAD="https://github.com/confluentinc/librdkafka/archive/refs/tags/v2.16.0.tar.gz"
LIBRDKAFKA_NAME=librdkafka-2.16.0.tar.gz
LIBRDKAFKA_SOURCE=librdkafka-2.16.0
LIBRDKAFKA_MD5SUM="6d2c62921241863d2bc95c58be27af6f"
# zstd
ZSTD_DOWNLOAD="https://github.com/facebook/zstd/releases/download/v1.5.7/zstd-1.5.7.tar.gz"
ZSTD_NAME=zstd-1.5.7.tar.gz
ZSTD_SOURCE=zstd-1.5.7
ZSTD_MD5SUM="780fc1896922b1bc52a4e90980cdda48"
# brotli
# Arrow builds and BE statically links this decoder, so a system library update
# cannot replace the vulnerable code fixed in 1.0.10 (CVE-2026-111497).
BROTLI_DOWNLOAD="https://github.com/google/brotli/archive/v1.0.10.tar.gz"
BROTLI_NAME="brotli-1.0.10.tar.gz"
BROTLI_SOURCE="brotli-1.0.10"
BROTLI_MD5SUM="830ce41562b2ec8cbdfe3f70111db0fa"
# flatbuffers
FLATBUFFERS_DOWNLOAD="https://github.com/google/flatbuffers/archive/v23.5.26.tar.gz"
FLATBUFFERS_NAME=flatbuffers-23.5.26.tar.gz
FLATBUFFERS_SOURCE=flatbuffers-23.5.26
FLATBUFFERS_MD5SUM="2ef00eaaa86ab5e9ad5eafe09c2e7b60"
# c-ares
CARES_DOWNLOAD="https://github.com/c-ares/c-ares/releases/download/cares-1_19_1/c-ares-1.19.1.tar.gz"
CARES_NAME="c-ares-1.19.1.tar.gz"
CARES_SOURCE=c-ares-1.19.1
CARES_MD5SUM="dafc5825a92dc907e144570e4e75a908"
# grpc
# grpc v1.55 and above require protobuf >= 22
GRPC_DOWNLOAD="https://github.com/grpc/grpc/archive/refs/tags/v1.54.3.tar.gz"
GRPC_NAME="grpc-v1.54.3.tar.gz"
GRPC_SOURCE=grpc-1.54.3
GRPC_MD5SUM="af00a2edeae0f02bb25917cc3473b7de"
# arrow
ARROW_DOWNLOAD="https://github.com/apache/arrow/archive/refs/tags/apache-arrow-24.0.0.tar.gz"
ARROW_NAME="apache-arrow-24.0.0.tar.gz"
ARROW_SOURCE="arrow-apache-arrow-24.0.0"
ARROW_MD5SUM="66c53bd00baa79034bd2ca167beea436"
# arrow-adbc
# One source tree, three artifacts: the driver manager (statically linked into
# doris_be), the JNI bridge (loaded by the FE adbc connector) and the SQLite
# driver (BE unit tests only, not shipped).
# Release "N" ships C/Go 1.12.0 and Java 0.24.0; the tag carries neither number.
ARROW_ADBC_DOWNLOAD="https://github.com/apache/arrow-adbc/archive/refs/tags/apache-arrow-adbc-24.tar.gz"
ARROW_ADBC_NAME="apache-arrow-adbc-24.tar.gz"
ARROW_ADBC_SOURCE="arrow-adbc-apache-arrow-adbc-24"
ARROW_ADBC_MD5SUM="2b2a18e95c33bdfd2bfa33a8b57c78d6"
# arrow-adbc Flight SQL driver, prebuilt.
# Upstream writes this driver in Go and publishes no bare shared library, so it is
# taken from the official release wheel (a zip) rather than adding a Go toolchain to
# the thirdparty build. It must come from the same release as the source tree above
# (C/Go 1.12.0): partition descriptors are driver-private bytes, so the FE and every
# BE have to load the very same file. ADBC tests only, not shipped -- same treatment
# as the SQLite driver. Linux only; the macOS wheels carry a .dylib no test looks for.
if [[ "$(uname -s)" == 'Linux' ]]; then
ARROW_ADBC_FLIGHTSQL_SOURCE="adbc_driver_flightsql"
if [[ "${MACHINE_TYPE}" == 'x86_64' ]]; then
ARROW_ADBC_FLIGHTSQL_DOWNLOAD="https://files.pythonhosted.org/packages/2b/5b/f8566aa6d05eb40225f5874f8f5f5e8b8b4a4ce866d817a5ff352f326302/adbc_driver_flightsql-1.12.0-py3-none-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl"
ARROW_ADBC_FLIGHTSQL_NAME="adbc_driver_flightsql-1.12.0-x86_64.zip"
ARROW_ADBC_FLIGHTSQL_MD5SUM="975a283dc72db0c646204e512b396e2a"
elif [[ "${MACHINE_TYPE}" == "aarch64" || "${MACHINE_TYPE}" == 'arm64' ]]; then
ARROW_ADBC_FLIGHTSQL_DOWNLOAD="https://files.pythonhosted.org/packages/57/45/6468fb425f3c0ae868c1ca9ab8e15cba76b59e941edd7e80e892e5476728/adbc_driver_flightsql-1.12.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl"
ARROW_ADBC_FLIGHTSQL_NAME="adbc_driver_flightsql-1.12.0-aarch64.zip"
ARROW_ADBC_FLIGHTSQL_MD5SUM="4ab31967910f9d2af20663b06412e6aa"
else
echo "no prebuilt adbc flightsql driver for ${MACHINE_TYPE}, skipping it"
ARROW_ADBC_FLIGHTSQL_SOURCE=""
fi
fi
# Abseil
ABSEIL_DOWNLOAD="https://github.com/abseil/abseil-cpp/releases/download/20250512.1/abseil-cpp-20250512.1.tar.gz"
ABSEIL_NAME="abseil-cpp-20250512.1.tar.gz"
ABSEIL_SOURCE=abseil-cpp-20250512.1
ABSEIL_MD5SUM="f225db90c7f3485f3bb65ab2aeae76fa"
# S2
S2_DOWNLOAD="https://github.com/google/s2geometry/archive/refs/tags/v0.10.0.tar.gz"
S2_NAME=s2geometry-0.10.0.tar.gz
S2_SOURCE=s2geometry-0.10.0
S2_MD5SUM="c68f3c5d326dde9255681b9201393a9f"
# bitshuffle
BITSHUFFLE_DOWNLOAD="https://github.com/kiyo-masui/bitshuffle/archive/0.5.1.tar.gz"
BITSHUFFLE_NAME=bitshuffle-0.5.1.tar.gz
BITSHUFFLE_SOURCE=bitshuffle-0.5.1
BITSHUFFLE_MD5SUM="b3bf6a9838927f7eb62214981c138e2f"
# croaringbitmap
CROARINGBITMAP_DOWNLOAD="https://github.com/RoaringBitmap/CRoaring/archive/refs/tags/v2.1.2.tar.gz"
CROARINGBITMAP_NAME=CRoaring-2.1.2.tar.gz
CROARINGBITMAP_SOURCE=CRoaring-2.1.2
CROARINGBITMAP_MD5SUM="419bfbafdf93e9a7e6cdc234454908fc"
# fmt
FMT_DOWNLOAD="https://github.com/fmtlib/fmt/archive/7.1.3.tar.gz"
FMT_NAME="fmt-7.1.3.tar.gz"
FMT_SOURCE="fmt-7.1.3"
FMT_MD5SUM="2522ec65070c0bda0ca288677ded2831"
# parallel-hashmap
PARALLEL_HASHMAP_DOWNLOAD="https://github.com/greg7mdp/parallel-hashmap/archive/refs/tags/v1.3.8.tar.gz"
PARALLEL_HASHMAP_NAME="parallel-hashmap-1.3.8.tar.gz"
PARALLEL_HASHMAP_SOURCE="parallel-hashmap-1.3.8"
PARALLEL_HASHMAP_MD5SUM="1b8130d0b4f656257ef654699bfbf941"
# orc
ORC_DOWNLOAD="https://archive.apache.org/dist/orc/orc-1.9.0/orc-1.9.0.tar.gz"
ORC_NAME="orc-1.9.0.tar.gz"
ORC_SOURCE=orc-1.9.0
ORC_MD5SUM="5dc1c91c4867e4519aab531ffc30fab7"
# jemalloc for arrow
JEMALLOC_ARROW_DOWNLOAD="https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2"
JEMALLOC_ARROW_NAME="jemalloc-5.3.0.tar.bz2"
JEMALLOC_ARROW_SOURCE="jemalloc-5.3.0"
JEMALLOC_ARROW_MD5SUM="09a8328574dab22a7df848eae6dbbf53"
# jemalloc for doris
JEMALLOC_DORIS_DOWNLOAD="https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2"
JEMALLOC_DORIS_NAME="jemalloc-5.3.0.tar.bz2"
JEMALLOC_DORIS_SOURCE="jemalloc-5.3.0"
JEMALLOC_DORIS_MD5SUM="09a8328574dab22a7df848eae6dbbf53"
# libunwind
LIBUNWIND_DOWNLOAD="https://github.com/libunwind/libunwind/releases/download/v1.8.3/libunwind-1.8.3.tar.gz"
LIBUNWIND_NAME="libunwind-1.8.3.tar.gz"
LIBUNWIND_SOURCE="libunwind-1.8.3"
LIBUNWIND_MD5SUM="13bc7b41462ac6ea157d350eaf6c1503"
# cctz
CCTZ_DOWNLOAD="https://github.com/google/cctz/archive/refs/tags/v2.5.tar.gz"
CCTZ_NAME="cctz-2.5.tar.gz"
CCTZ_SOURCE="cctz-2.5"
CCTZ_MD5SUM="cae0ee4a5da5399d591e770b7515a46d"
# datatables, bootstrap 3 and jQuery 3
# The origin download url is always changing: https://datatables.net/download/builder?bs-3.3.7/jq-3.3.1/dt-1.10.25
# So we put it in our own http server.
# If someone can offer an official url for DataTables, please update this.
DATATABLES_DOWNLOAD="https://github.com/apache/doris-thirdparty/releases/download/datatables-1.12.1/DataTables.zip"
DATATABLES_NAME="DataTables.zip"
DATATABLES_SOURCE="DataTables-1.12.1"
DATATABLES_MD5SUM="a3dd92a2a8b7254443e102a43036d743"
# bootstrap table js
BOOTSTRAP_TABLE_JS_DOWNLOAD="https://unpkg.com/bootstrap-table@1.17.1/dist/bootstrap-table.min.js"
BOOTSTRAP_TABLE_JS_NAME="bootstrap-table.min.js"
BOOTSTRAP_TABLE_JS_FILE="bootstrap-table.min.js"
BOOTSTRAP_TABLE_JS_MD5SUM="6cc9c41eaf7e81e54e220061cc9c0432"
# bootstrap table css
BOOTSTRAP_TABLE_CSS_DOWNLOAD="https://unpkg.com/bootstrap-table@1.17.1/dist/bootstrap-table.min.css"
BOOTSTRAP_TABLE_CSS_NAME="bootstrap-table.min.css"
BOOTSTRAP_TABLE_CSS_FILE="bootstrap-table.min.css"
BOOTSTRAP_TABLE_CSS_MD5SUM="23389d4456da412e36bae30c469a766a"
# aws sdk
AWS_SDK_DOWNLOAD="https://github.com/aws/aws-sdk-cpp/archive/refs/tags/1.11.221.tar.gz"
AWS_SDK_NAME="aws-sdk-cpp-1.11.221.tar.gz"
AWS_SDK_SOURCE="aws-sdk-cpp-1.11.221"
AWS_SDK_MD5SUM="95ea128da58829117a544b092bc39033"
# tsan_header
TSAN_HEADER_DOWNLOAD="https://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=libsanitizer/include/sanitizer/tsan_interface_atomic.h;hb=refs/heads/releases/gcc-7"
# Not every CI runner can reach gcc.gnu.org, and download_func() spends the first
# of its two passes re-checking the file that is already on disk, so an
# unreachable host costs a single attempt and then fails the build. The gcc-7
# branch is closed, so this mirror serves a byte identical file.
TSAN_HEADER_FALLBACK_DOWNLOAD="https://raw.githubusercontent.com/gcc-mirror/gcc/releases/gcc-7/libsanitizer/include/sanitizer/tsan_interface_atomic.h"
TSAN_HEADER_NAME="tsan_interface_atomic.h"
TSAN_HEADER_FILE="tsan_interface_atomic.h"
TSAN_HEADER_MD5SUM="d72679bea167d6a513d959f5abd149dc"
# lzma
LZMA_DOWNLOAD="https://github.com/kobolabs/liblzma/archive/refs/heads/master.zip"
LZMA_NAME="liblzma-master.zip"
LZMA_SOURCE="liblzma-master"
LZMA_MD5SUM="ef11f2fbbfa6893b629f207a32bf730e"
# xml2
XML2_DOWNLOAD="https://gitlab.gnome.org/GNOME/libxml2/-/archive/v2.9.10/libxml2-v2.9.10.tar.gz"
XML2_NAME="libxml2-v2.9.10.tar.gz"
XML2_SOURCE="libxml2-v2.9.10"
XML2_MD5SUM="b18faee9173c3378c910f6d7d1493115"
# idn
IDN_DOWNLOAD="https://ftp.nluug.nl/pub/gnu/libidn/libidn-1.38.tar.gz"
IDN_NAME="libidn-1.38.tar.gz"
IDN_SOURCE="libidn-1.38"
IDN_MD5SUM="718ff3700dd71f830c592ebe97249193"
# gsasl
GSASL_DOWNLOAD="https://ftp.nluug.nl/pub/gnu/gsasl/libgsasl-1.8.0.tar.gz"
GSASL_NAME="libgsasl-1.8.0.tar.gz"
GSASL_SOURCE="libgsasl-1.8.0"
GSASL_MD5SUM="5dbdf859f6e60e05813370e2b193b92b"
# krb5
KRB5_DOWNLOAD="https://kerberos.org/dist/krb5/1.19/krb5-1.19.tar.gz"
KRB5_NAME="krb5-1.19.tar.gz"
KRB5_SOURCE="krb5-1.19"
KRB5_MD5SUM="aaf18447a5a014aa3b7e81814923f4c9"
# hdfs3
HDFS3_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/libhdfs3-v2.3.9.tar.gz"
HDFS3_NAME="doris-thirdparty-libhdfs3-v2.3.9.tar.gz"
HDFS3_SOURCE="doris-thirdparty-libhdfs3-v2.3.9"
HDFS3_MD5SUM="b3eaa03e5b184521e5ad5bf6cabea97e"
#libdivide
LIBDIVIDE_DOWNLOAD="https://github.com/ridiculousfish/libdivide/archive/refs/tags/v5.2.0.tar.gz"
LIBDIVIDE_NAME="libdivide-5.2.0.tar.gz"
LIBDIVIDE_SOURCE="libdivide-5.2.0"
LIBDIVIDE_MD5SUM="4ba77777192c295d6de2b86d88f3239a"
#pdqsort
PDQSORT_DOWNLOAD="https://raw.githubusercontent.com/orlp/pdqsort/b1ef26a55cdb60d236a5cb199c4234c704f46726/pdqsort.h"
PDQSORT_NAME="pdqsort.h"
PDQSORT_FILE="pdqsort.h"
PDQSORT_MD5SUM="af28f79d5d7d7a5486f54d9f1244c2b5"
#timsort
TIMSORT_DOWNLOAD="https://raw.githubusercontent.com/timsort/cpp-TimSort/3.x.y/include/gfx/timsort.hpp"
TIMSORT_NAME="timsort.hpp"
TIMSORT_FILE="timsort.hpp"
TIMSORT_MD5SUM="000db489fb9c39a1cfb64a034f16b04d"
# benchmark
BENCHMARK_DOWNLOAD="https://github.com/google/benchmark/archive/refs/tags/v1.8.0.tar.gz"
BENCHMARK_NAME=benchmark-v1.8.0.tar.gz
BENCHMARK_SOURCE=benchmark-1.8.0
BENCHMARK_MD5SUM="8ddf8571d3f6198d37852bcbd964f817"
# xsimd
# for arrow-24.0.0, if arrow upgrade, this version may also need to be changed
XSIMD_DOWNLOAD="https://github.com/xtensor-stack/xsimd/archive/refs/tags/14.0.0.tar.gz"
XSIMD_NAME="14.0.0.tar.gz"
XSIMD_SOURCE=xsimd-14.0.0
XSIMD_MD5SUM="75c0d34cf7011924ba19978076c76dc1"
# simdjson
SIMDJSON_DOWNLOAD="https://github.com/simdjson/simdjson/archive/refs/tags/v3.11.6.tar.gz"
SIMDJSON_NAME=simdjson-3.11.6.tar.gz
SIMDJSON_SOURCE=simdjson-3.11.6
SIMDJSON_MD5SUM="e7d9c814a4fdd6e47119ce5cf4240f4e"
# simdutf
SIMDUTF_DOWNLOAD="https://github.com/simdutf/simdutf/archive/refs/tags/v9.2.0.tar.gz"
SIMDUTF_NAME=simdutf-9.2.0.tar.gz
SIMDUTF_SOURCE=simdutf-9.2.0
SIMDUTF_MD5SUM="abeae9267c1c1caf183b8d24a39c0147"
# nlohmann_json
NLOHMANN_JSON_DOWNLOAD="https://github.com/nlohmann/json/archive/refs/tags/v3.10.1.tar.gz"
NLOHMANN_JSON_NAME=json-3.10.1.tar.gz
NLOHMANN_JSON_SOURCE=json-3.10.1
NLOHMANN_JSON_MD5SUM="7b369d567afc0dffdcf5800fd9abb836"
# Google Cloud C++ OAuth2 authentication library. Only the auth/common REST
# components are built; the Google Cloud Storage SDK is intentionally excluded.
GOOGLE_CLOUD_CPP_DOWNLOAD="https://github.com/googleapis/google-cloud-cpp/archive/refs/tags/v2.45.0.tar.gz"
GOOGLE_CLOUD_CPP_FALLBACK_DOWNLOAD="https://codeload.github.com/googleapis/google-cloud-cpp/tar.gz/refs/tags/v2.45.0"
GOOGLE_CLOUD_CPP_NAME="google-cloud-cpp-2.45.0.tar.gz"
GOOGLE_CLOUD_CPP_SOURCE="google-cloud-cpp-2.45.0"
GOOGLE_CLOUD_CPP_MD5SUM="3cc131764cd9009c20407d7b67afcf6e"
# libbacktrace
LIBBACKTRACE_DOWNLOAD="https://codeload.github.com/ianlancetaylor/libbacktrace/zip/2446c66076480ce07a6bd868badcbceb3eeecc2e"
LIBBACKTRACE_NAME=libbacktrace-2446c66076480ce07a6bd868badcbceb3eeecc2e.zip
LIBBACKTRACE_SOURCE=libbacktrace-2446c66076480ce07a6bd868badcbceb3eeecc2e
LIBBACKTRACE_MD5SUM="6c79a8012870a24610c0d9c3621b23fe"
# sse2noen
SSE2NEON_DOWNLOAD="https://github.com/DLTcollab/sse2neon/archive/refs/tags/v1.6.0.tar.gz"
SSE2NEON_NAME=sse2neon-1.6.0.tar.gz
SSE2NEON_SOURCE=sse2neon-1.6.0
SSE2NEON_MD5SUM="dce28eb6a78f45bf98740d5fad73febb"
# xxhash
XXHASH_DOWNLOAD="https://github.com/Cyan4973/xxHash/archive/refs/tags/v0.8.3.tar.gz"
XXHASH_NAME=xxHash-0.8.3.tar.gz
XXHASH_SOURCE=xxHash-0.8.3
XXHASH_MD5SUM="599804eb9555e51c05f1b821f9212a07"
# concurrentqueue
CONCURRENTQUEUE_DOWNLOAD="https://github.com/cameron314/concurrentqueue/archive/refs/tags/v1.0.3.tar.gz"
CONCURRENTQUEUE_NAME=concurrentqueue-1.0.3.tar.gz
CONCURRENTQUEUE_SOURCE=concurrentqueue-1.0.3
CONCURRENTQUEUE_MD5SUM="118e5bb661b567634647312991e10222"
# fast_float
FAST_FLOAT_DOWNLOAD="https://github.com/fastfloat/fast_float/archive/refs/tags/v3.9.0.tar.gz"
FAST_FLOAT_NAME=fast_float-3.9.0.tar.gz
FAST_FLOAT_SOURCE=fast_float-3.9.0
FAST_FLOAT_MD5SUM="5656b0d8b150a3b157cfb092d214f6ea"
# libhdfs
HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.4-for-doris.tar.gz"
HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.4-for-doris.tar.gz"
HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.4-for-doris"
HADOOP_LIBS_3_4_MD5SUM="729a7ef9126f65255d1da124a88ffd50"
# AvxToNeon
AVX2NEON_DOWNLOAD="https://github.com/kunpengcompute/AvxToNeon/archive/refs/tags/v1.0.0.tar.gz"
AVX2NEON_NAME=v1.0.0.tar.gz
AVX2NEON_SOURCE=AvxToNeon-1.0.0
AVX2NEON_MD5SUM="692d0e0f8b885a86ebc5172a9d8ee8db"
# libdeflate
LIBDEFLATE_DOWNLOAD="https://github.com/ebiggers/libdeflate/archive/refs/tags/v1.19.tar.gz"
LIBDEFLATE_NAME=libdeflate-1.19.tar.gz
LIBDEFLATE_SOURCE=libdeflate-1.19
LIBDEFLATE_MD5SUM="c69e9193d2975a729068ffa862c81fb6"
# streamvbyte
STREAMVBYTE_DOWNLOAD="https://github.com/lemire/streamvbyte/archive/refs/tags/v1.0.0.tar.gz"
STREAMVBYTE_NAME=streamvbyte-1.0.0.tar.gz
STREAMVBYTE_SOURCE=streamvbyte-1.0.0
STREAMVBYTE_MD5SUM="f334219db5a832b6dae3589a56a29563"
# jsoncpp
JSONCPP_DOWNLOAD="https://github.com/open-source-parsers/jsoncpp/archive/refs/tags/1.9.5.tar.gz"
JSONCPP_NAME="1.9.5.tar.gz"
JSONCPP_SOURCE="jsoncpp-1.9.5"
JSONCPP_MD5SUM="d6c8c609f2162eff373db62b90a051c7"
# libuuid
LIBUUID_DOWNLOAD="https://downloads.sourceforge.net/project/libuuid/libuuid-1.0.3.tar.gz"
LIBUUID_NAME="libuuid-1.0.3.tar.gz"
LIBUUID_SOURCE="libuuid-1.0.3"
LIBUUID_MD5SUM="d44d866d06286c08ba0846aba1086d68"
# ali
ALI_SDK_DOWNLOAD="https://github.com/aliyun/aliyun-openapi-cpp-sdk/archive/refs/tags/1.36.1586.tar.gz"
ALI_SDK_NAME="1.36.1586.tar.gz"
ALI_SDK_SOURCE="aliyun-openapi-cpp-sdk-1.36.1586"
ALI_SDK_MD5SUM="14623ee8e87c89615477d9bf8cb30bea"
# base64
BASE64_DOWNLOAD="https://github.com/aklomp/base64/archive/refs/tags/v0.5.2.tar.gz"
BASE64_NAME="v0.5.2.tar.gz"
BASE64_SOURCE="base64-0.5.2"
BASE64_MD5SUM="49e5a6c98bd0192aedd16c16eec39974"
# azure blob
AZURE_DOWNLOAD="https://github.com/Azure/azure-sdk-for-cpp/archive/azure-core_1.16.0.tar.gz"
AZURE_NAME="azure-core_1.16.0.tar.gz"
AZURE_SOURCE="azure-sdk-for-cpp-azure-core_1.16.0"
AZURE_MD5SUM="0fb23eab695dc24b94456bf461e6af3b"
# libdragonbox for faster double/float to string
DRAGONBOX_DOWNLOAD="https://github.com/jk-jeon/dragonbox/archive/refs/tags/1.1.3.tar.gz"
DRAGONBOX_NAME=dragonbox-1.1.3.tar.gz
DRAGONBOX_SOURCE=dragonbox-1.1.3
DRAGONBOX_MD5SUM="889dc00db9612c6949a4ccf8115e0e6a"
# icu
ICU_DOWNLOAD="https://github.com/unicode-org/icu/archive/refs/tags/release-69-1.tar.gz"
ICU_NAME=release-69-1.tar.gz
ICU_SOURCE=icu-release-69-1
ICU_MD5SUM="135125f633864285d637db5c01e0388b"
# mecab-ipadic (UTF-8) for the Japanese kuromoji analyzer
MECAB_IPADIC_DOWNLOAD="https://github.com/lindera/mecab-ipadic/archive/refs/tags/2.7.0-20250920.tar.gz"
MECAB_IPADIC_NAME="mecab-ipadic-2.7.0-20250920.tar.gz"
MECAB_IPADIC_SOURCE="mecab-ipadic-2.7.0-20250920"
MECAB_IPADIC_MD5SUM="a95c409f12f1023fce8ef91f991ef042"
# jindofs
JINDOFS_DOWNLOAD="https://github.com/apache/doris-thirdparty/releases/download/alibabacloud-jindodata-releases/jindofs-6.10.4-libs-0.1.tar.gz"
JINDOFS_NAME=jindofs-6.10.4-libs-0.1.tar.gz
JINDOFS_SOURCE=jindofs-6.10.4-libs-0.1
JINDOFS_MD5SUM="bd30b4c5fe97c4367eeb3bb228b317d9"
# juicefs
JUICEFS_DOWNLOAD="${DORIS_THIRDPARTY_REPOSITORY_URL%/}/juicefs-hadoop-1.3.1.jar"
JUICEFS_FALLBACK_DOWNLOAD="https://repo1.maven.org/maven2/io/juicefs/juicefs-hadoop/1.3.1/juicefs-hadoop-1.3.1.jar"
JUICEFS_NAME=juicefs-hadoop-1.3.1.jar
JUICEFS_SOURCE=
JUICEFS_MD5SUM="f374dfbfbdc4b83417cfea78a6728c54"
# pugixml
PUGIXML_DOWNLOAD="https://github.com/zeux/pugixml/releases/download/v1.15/pugixml-1.15.tar.gz"
PUGIXML_NAME=pugixml-1.15.tar.gz
PUGIXML_SOURCE=pugixml-1.15
PUGIXML_MD5SUM="3b894c29455eb33a40b165c6e2de5895"
# lance-c
# Includes lance-c #98: bounded runtime BTree segment selection with decoded-memory guards.
LANCE_C_DOWNLOAD="https://codeload.github.com/lance-format/lance-c/tar.gz/a51ab47a9c814a746cb1714c99fad93dd3a188b8"
LANCE_C_NAME="lance-c-a51ab47a9c814a746cb1714c99fad93dd3a188b8.tar.gz"
LANCE_C_SOURCE="lance-c-a51ab47a9c814a746cb1714c99fad93dd3a188b8"
LANCE_C_MD5SUM="d1282cff3d63acfc0450db1f00d6ab6f"
# paimon-rust
PAIMON_RUST_DOWNLOAD="https://github.com/apache/paimon-rust/archive/refs/tags/v0.4.0-rc1.tar.gz"
PAIMON_RUST_NAME="paimon-rust-v0.4.0-rc1.tar.gz"
PAIMON_RUST_SOURCE="paimon-rust-0.4.0-rc1"
PAIMON_RUST_MD5SUM="23cd0634692108debf77a26ef4dc20a7"
# all thirdparties which need to be downloaded is set in array TP_ARCHIVES
export TP_ARCHIVES=(
'MECAB_IPADIC'
'LIBEVENT'
'OPENSSL'
'THRIFT'
'PROTOBUF'
'GFLAGS'
'GLOG'
'GTEST'
'RAPIDJSON'
'SNAPPY'
'GPERFTOOLS'
'ZLIB'
'CRC32C'
'LZ4'
'BZIP'
'LZO2'
'CURL'
'RE2'
'HYPERSCAN'
'RAGEL'
'BOOST'
'MYSQL'
'ODBC'
'LEVELDB'
'BRPC'
'ROCKSDB'
'CYRUS_SASL'
'LIBRDKAFKA'
'FLATBUFFERS'
'ARROW'
'ARROW_ADBC'
'BROTLI'
'ZSTD'
'ABSEIL'
'S2'
'BITSHUFFLE'
'CROARINGBITMAP'
'FMT'
'PARALLEL_HASHMAP'
'ORC'
'CARES'
'GRPC'
'JEMALLOC_ARROW'
'JEMALLOC_DORIS'
'LIBUNWIND'
'CCTZ'
'DATATABLES'
'BOOTSTRAP_TABLE_JS'
'BOOTSTRAP_TABLE_CSS'
'TSAN_HEADER'
'AWS_SDK'
'LZMA'
'XML2'
'IDN'
'GSASL'
'KRB5'
'HDFS3'
'LIBDIVIDE'
'PDQSORT'
'TIMSORT'
'BENCHMARK'
'XSIMD'
'SIMDJSON'
'SIMDUTF'
'NLOHMANN_JSON'
'GOOGLE_CLOUD_CPP'
'LIBBACKTRACE'
'SSE2NEON'
'XXHASH'
'CONCURRENTQUEUE'
'FAST_FLOAT'
'HADOOP_LIBS_3_4'
'AVX2NEON'
'LIBDEFLATE'
'STREAMVBYTE'
'JSONCPP'
'LIBUUID'
'ALI_SDK'
'BASE64'
'AZURE'
'DRAGONBOX'
'ICU'
'JINDOFS'
'JUICEFS'
'PUGIXML'
'LANCE_C'
'PAIMON_RUST'
)
# This variable is unset on macOS; callers may source this file with nounset.
if [[ -n "${ARROW_ADBC_FLIGHTSQL_SOURCE:-}" ]]; then
read -r -a TP_ARCHIVES <<<"${TP_ARCHIVES[*]} ARROW_ADBC_FLIGHTSQL"
export TP_ARCHIVES
fi
if [[ "$(uname -s)" == 'Darwin' ]]; then
#binutils
BINUTILS_DOWNLOAD='https://ftp.nluug.nl/pub/gnu/binutils/binutils-2.39.tar.gz'
BINUTILS_NAME=binutils-2.39.tar.gz
BINUTILS_SOURCE=binutils-2.39
BINUTILS_MD5SUM='ab6825df57514ec172331e988f55fc10'
#gettext
GETTEXT_DOWNLOAD='https://ftp.nluug.nl/pub/gnu/gettext/gettext-0.21.tar.gz'
GETTEXT_NAME='gettext-0.21.tar.gz'
GETTEXT_SOURCE='gettext-0.21'
GETTEXT_MD5SUM='28b1cd4c94a74428723ed966c38cf479'
read -r -a TP_ARCHIVES <<<"${TP_ARCHIVES[*]} BINUTILS GETTEXT"
export TP_ARCHIVES
fi
Reference in new issue View Git Blame Copy Permalink