10 March 2023

Fuzzing

  • Check libFuzzer on clang.
  • Check Radamsa again.

Podman on RHEL 8

  • Need crun instead of runc.
  • Need to set /etc/subuid and /etc/subgid.

ClickHouse / Zookeeper replication lag

  • Possibly caused by lack of NTP on servers.
  • Problem seems to be gone after setting up NTP on the servers.

15 March 2023

ClickHouse has really nice colorful logs. They’re randomly generated to have a random color compontent, but consistent brightness. The code for that can be found here.

19 March 2023

Making a custom colorspace

// TODO: Write this

21 March 2023

Static zstd binary

The motivation

When deploying applications and dependencies to customer VMs, it is useful to reduce the number of files and the total size you need to upload.

While we could use something like gzip for this, it would be better to use zstd because it manages to compress better and faster. And even after deployment, we can use it in our tools and scripts.

As the result of this project, I want to have a single executable file that I can copy to a VM to compress and decompress data. I don’t want to deal with glibc version incompatibilities.

And I don’t want to need root access to install any dependencies, this should work on any VM we are given, regardless of permissions.

The solution

A good candidate for doing this is musl libc, which is designed to be statically compiled. Alpine Linux uses musl libc by default, and makes this really easy to do. We can just use a Podman container and build a static zstd binary inside it.

Aside from working on zstd-compressed files, the zstd tool can also use the lz4, gz and xz/lzma formats. This makes it a pretty good “universal” compression tool, and having all that in a single binary is very convenient.

Below is the script I used to compile a static zstd binary on Alpine Linux.

apk update
apk add wget
apk add make
apk add gcc
apk add musl-dev

apk add lz4-dev
apk add lz4-static
apk add zlib-dev
apk add zlib-static
apk add xz-dev
apk add xz-static

ZSTD_VERSION=1.5.4
URL="https://github.com/facebook/zstd/releases/download"
URL="${URL}/v${ZSTD_VERSION}/zstd-${ZSTD_VERSION}.tar.gz"

wget -O zstd.tar.gz "${URL}"
tar -xzf zstd.tar.gz
cd zstd-${ZSTD_VERSION}

export CFLAGS="-flto"
export LDLAGS="-flto -static -static-libgcc -static-libstdc++"
export HAVE_LZ4=1
export HAVE_ZLIB=1
export HAVE_LZMA=1
make zstd-release

cp programs/zstd /output/path/zstd

The results

And the result? A single 2.6 MB file that can compress/decompress any format we care about using.

$ file ./zstd ./zstd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped

$ ls -lah ./zstd -rwxrwxrwx 1 root root 2.6M Mar 21 13:28 ./zstd

$ ./zstd –help

–format=zstd Compress files to the .zst format. [Default] –format=gzip Compress files to the .gz format. –format=xz Compress files to the .xz format. –format=lzma Compress files to the .lzma format. –format=lz4 Compress files to the .lz4 format.

31 March 2023

  • To visualize and plot 1D data, you can use PulseView from sigrok, or GTKWave.
    • The format most of those waveform/logic viewers use is called VCD, or Value Change Dump.
    • Should be possible to make WAV files too. Both PulseView and Audacity can open those.