The motivation -------------- When deploying applications and dependencies to customer VMs, it is useful to reduce the number of files and the total size you need to upload. While we could use something like gzip for this, it would be better to use zstd because it manages to compress better and faster. And even after deployment, we can use it in our tools and scripts. As the result of this project, I want to have a single executable file that I can copy to a VM to compress and decompress data. I don't want to deal with glibc version incompatibilities. And I don't want to need root access to install any dependencies, this should work on any VM we are given, regardless of permissions. The solution ------------ A good candidate for doing this is musl libc, which is designed to be statically compiled. Alpine Linux uses musl libc by default, and makes this really easy to do. We can just use a Podman container and build a static zstd binary inside it. Aside from working on zstd-compressed files, the zstd tool can also use the lz4, gz and xz/lzma formats. This makes it a pretty good "universal" compression tool, and having all that in a single binary is very convenient. Below is the script I used to compile a static zstd binary on Alpine Linux. apk update apk add wget apk add make apk add gcc apk add musl-dev apk add lz4-dev apk add lz4-static apk add zlib-dev apk add zlib-static apk add xz-dev apk add xz-static ZSTD_VERSION=1.5.4 URL="https://github.com/facebook/zstd/releases/download" URL="${URL}/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz" wget -O zstd.tar.gz "${URL}" tar -xzf zstd.tar.gz cd zstd-${ZSTD_VERSION} export CFLAGS="-flto" export LDLAGS="-flto -static -static-libgcc -static-libstdc++" export HAVE_LZ4=1 export HAVE_ZLIB=1 export HAVE_LZMA=1 make zstd-release cp programs/zstd /output/path/zstd The results ----------- And the result? A single 2.6 MB file that can compress/decompress any format we care about using. $ file ./zstd ./zstd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped $ ls -lah ./zstd -rwxrwxrwx 1 root root 2.6M Mar 21 13:28 ./zstd $ ./zstd --help --format=zstd Compress files to the `.zst` format. [Default] --format=gzip Compress files to the `.gz` format. --format=xz Compress files to the `.xz` format. --format=lzma Compress files to the `.lzma` format. --format=lz4 Compress files to the `.lz4` format. -- Leo
[2023-03-21] Static zstd binary
Reading time: about 1 minute