version `GLIBC_2.38' not found, and the release check that stops it

./qdrant: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found

If you have seen that, you already know the shape of the problem: a binary that runs perfectly on the machine that built it, and cannot start at all on a slightly older one. What is less obvious is that you can hit it three completely different ways without changing a line of your own code, and that none of the usual release gates will catch any of them.

We ship prebuilt binaries for a couple of dozen database engines across five platforms. In one month we got bitten three times.

Why glibc breaks in one direction only

glibc symbols are versioned. When a binary uses a function, the linker records the minimum glibc version that provides the exact behavior it compiled against, as a symbol version like GLIBC_2.38. At startup the dynamic linker checks those requirements against the glibc actually installed.

The compatibility runs one way. A binary built on an older glibc runs on newer systems, because newer glibc keeps old symbol versions around. A binary built on a newer glibc does not run on older systems, because the symbol version it names simply does not exist there.

So your build machine's glibc becomes a floor on every machine that can run your output. Build on Ubuntu 24.04 (glibc 2.39) and you have silently dropped Ubuntu 22.04 (glibc 2.35), which is still an LTS release plenty of people run.

The failure is total, not partial. The process never reaches main. There is no graceful degradation and no useful diagnostic beyond the line above.

Three ways it reaches you

1. The upstream project's own binary requires it. Official SQLite binaries for Linux require GLIBC 2.38 or newer. Not a mistake on their part, just a build environment choice, and it makes them unusable on Ubuntu 22.04 and older. Our fix was to stop shipping the official Linux binary and build SQLite from the amalgamation source inside ubuntu:20.04, which gives a 2.31 floor. Nothing about the resulting binary is worse; it simply has a lower floor.

2. The upstream project changes their build between releases. Qdrant 1.16.3's linux-gnu tarball needed only GLIBC_2.34. Version 1.18.3 of the same asset needed GLIBC_2.38. No announcement, no major version bump, and the only visible symptom was that half our test matrix went red: Ubuntu 24.04 (glibc 2.39) was fine, Ubuntu 22.04 was not.

The fix there was different. Qdrant also publishes a linux-musl tarball, which is statically linked and references no glibc symbols at all. Static binaries pass this problem trivially, which is worth remembering as an option whenever a project offers both.

3. Your base image moves underneath you. This is the sneakiest one. We extracted CouchDB from the official couchdb Docker image, taking /opt/couchdb but not the system libraries the bundled Erlang runtime links against. That image tracks Debian stable, and at 3.5.2 it moved from bookworm to trixie.

So the extracted tree silently inherited a different distro's libraries. On Ubuntu 22.04 the Erlang VM needed GLIBC_2.38 and never started. On Ubuntu 24.04 it started and then died differently, because crypto.so needed OPENSSL_3.4.0:

{application_start_failure,config,{undef,{crypto,info_fips,[]}}}

Two distinct failures from one upstream base-image bump. And because we only logged "start timeout, then ECONNREFUSED," neither cause was visible until someone read the engine's own log.

That one was fixed by building from the project's apt repository pinned to jammy inside ubuntu:22.04, so the toolchain is pinned to our oldest supported target rather than to whatever a third party's base image happens to be this month.

The part worth stealing: check what your artifacts actually require

All three shipped green. Every existing gate passed. They only failed two repositories downstream, in a consumer's Ubuntu 22.04 CI, because nothing in the release pipeline ever looked at what the binaries required.

That is the real bug. The individual glibc problems are ordinary; the absence of a check is what let them all reach users.

The check is not complicated. For every Linux archive in a release, extract it, find every ELF file, read the highest GLIBC_x.y.z symbol version referenced, and fail if anything exceeds your declared floor:

# highest glibc symbol version a single ELF file requires
readelf -V "$file" 2>/dev/null \
  | grep -o 'GLIBC_[0-9.]*' \
  | sort -V \
  | tail -1

objdump -T works as a fallback where readelf is unavailable. A few details make the difference between a check that works and one people disable:

Find ELF files by magic bytes, not by extension. Binaries in these archives are often extensionless, and plenty of .so files live in subdirectories you did not expect.

Skip non-ELF payloads gracefully. JVM engines ship jars with nothing to inspect. If your check errors on those, it becomes noise and gets bypassed.

Static binaries pass trivially. musl, Go, and Zig builds reference no glibc symbols, so they return nothing and pass. That is correct, not a gap.

Put the floor in one constant. Ours is GLIBC_FLOOR="2.35", matching Ubuntu 22.04, the oldest target we support and the base image our build containers use. One constant, one place to change it when the support window moves.

Skip macOS and Windows archives. There is no glibc there, and trying to inspect them produces confusing failures.

We wired that into every release workflow. It would have caught all three incidents above at build time rather than in someone else's CI.

The general version

Anywhere you redistribute a compiled artifact, the build environment is part of the artifact's contract, and it is a part nobody declares. It gets inherited from a base image, from a CI runner's default, or from whatever a third party built their release on.

So state your floor explicitly, build on it deliberately, and add a gate that reads the requirement out of the artifact rather than trusting that it matches. The check costs seconds and it fails loudly in the one place you can still do something about it.

If you only take one thing: the glibc version of your build machine is a compatibility decision you are making whether or not you know it.

We run this check across every engine we ship at Layerbase, because "works on the build machine" turned out to be a guarantee about nothing at all.