The test-msan lane was red repo-wide: the Dockerfile.msan image build could
no longer install clang-22 from apt.llvm.org's noble-22 channel, failing with
"E: Unable to locate package clang-22 / libclang-rt-22-dev / llvm-22" after
the 5 apt retries, so the image never built and every PR's required test-msan
check could not go green (#2123).
apt.llvm.org keeps only the last few majors for noble and rotates newer ones
in and out; the -22 channel's package index was empty when CI fetched it (the
channel has since flapped back, confirming the transience). Rather than stay on
that volatile channel, repin to clang-21 - the highest well-established stable
noble channel apt.llvm.org currently serves (verified installable on both
amd64 and arm64):
- apt source noble-22 -> noble-21, clang-22/libclang-rt-22-dev/llvm-22 ->
clang-21/libclang-rt-21-dev/llvm-21, and the clang/clang++ symlinks
- MSan runtimes clone llvmorg-22.1.0 -> llvmorg-21.1.8 (matches the apt
clang 21.1.8, keeping the libc++/libc++abi/libunwind build at the
compiler's major, as the layer requires)
- MSAN_SYMBOLIZER_PATH /usr/lib/llvm-22 -> /usr/lib/llvm-21
Verified in an ubuntu:noble container that clang-21/libclang-rt-21-dev/llvm-21
install cleanly, the clang/clang++ symlinks resolve, libclang_rt.msan-*.a and
llvm-symbolizer are present, and the full image builds via Colima.
Infra only; the MSan lane's other files (docker-compose.yml, scripts/msan.sh)
use the version-agnostic clang/clang++ symlinks and need no change.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-msan lane dies whenever the buildx layer cache misses and the
builder's resolver hiccups. PR-CI runs 33816056229 and 33818108315 both
failed in "Build MSan image (cached layers)" with:
#7 26.77 Could not resolve 'apt.llvm.org'
#7 27.34 W: Failed to fetch http://apt.llvm.org/noble/dists/\
llvm-toolchain-noble-22/InRelease Could not resolve 'apt.llvm.org'
ERROR: failed to build: failed to solve: process "/bin/sh -c apt-get \
update && ..." did not complete successfully
The wget of the signing key from that SAME host had succeeded seconds
earlier in the same layer, so this is a transient DNS failure inside
buildkit, not a wrong URL or a dead mirror. 2 of the last 15 PR runs.
Every network-touching step in Dockerfile.msan now runs under a small
POSIX-sh helper written into the image before the first layer that needs
it: /usr/local/bin/retry makes 5 attempts with growing backoff
(5s/10s/20s/40s) and still exits non-zero once they are exhausted, so a
genuine breakage keeps failing the build instead of being swallowed.
Wrapped: both apt-get update/install pairs (also given
-o Acquire::Retries=3), the wget of the LLVM signing key, and the
llvm-project and zlib clones. Two shapes needed care:
- the key is fetched with `wget -O <file>` instead of `-qO- > <file>`,
because a redirect is opened once for all attempts and a partial write
from a failed attempt would be prepended to a later successful one;
- each clone clears its destination first, or a half-finished clone makes
every later attempt fail on "destination path already exists".
The pinned base digest, the pinned llvmorg-22.1.0 / v1.3.1 tags,
--no-install-recommends and the layer structure are unchanged; on a
healthy network the image is byte-for-byte the same work as before.
Verified on Colima:
- `docker buildx build --check -f test-infrastructure/Dockerfile.msan
test-infrastructure/` -> "Check complete, no warnings found."
- scratch image = FROM + helper + the real apt.llvm.org layer, built
--no-cache: layer completes, clang-22/llvm-22 installed.
- scratch image where the wrapped command fails twice then succeeds:
attempts 1 and 2 logged with 5s/10s backoff, attempt 3 succeeds,
build exit 0.
- scratch image where the wrapped command always fails: 5 attempts,
"retry: giving up after 5 attempts", retry_exit=1, build exit 1.
The full MSan image was deliberately not built locally (it compiles
libc++ with MSan); CI builds it on the branch.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every lane here was either asserting coverage it did not have, or was
reachable only by bypassing the entry point that is supposed to define
the ladder.
TSan: no exclusions any more. The three suites the Makefile documented
as excluded are back. daemon_ipc and daemon_frontend no longer reproduce
the harness race and thread leaks they were excluded for. daemon_runtime
did not deadlock as the comment claimed -- it reported a real production
data race on the log sink, fixed separately. Excluding a suite from a
sanitizer lane hides exactly the class of bug the lane exists to find,
so the comment block now records what was actually true rather than what
was assumed.
TSAN_OPTIONS gains report_thread_leaks=0. This disables the thread-
HYGIENE check only; race detection is untouched. Several daemon fixtures
fork after the process has gone multi-threaded, and in the forked child
TSan sees the parent's already-finished threads as never-joined even
where the fixture joins them. It fires on macOS and not Linux, i.e. it
tracks fork semantics rather than anything about this code. The
alternative was dropping whole suites, which costs real race coverage;
this costs none.
UBSan: tre.o no longer builds with -fno-sanitize=alignment. Alignment
was switched off for a vendored regex engine that ships in the product,
which is where the check is least redundant, not most.
LSan on macOS: new test-lsan target and test-lsan-macos CI leg. LSan is
on by default under ASan on Linux, so the Linux legs have always had
leak coverage. On macOS it is off by default and Apple's clang refuses
to enable it outright, so that platform had none at all. Apple's refusal
is not a darwin limitation -- upstream LLVM supports LSan on darwin/
arm64. The lane is the ordinary ASan suite built with Homebrew LLVM and
run with detect_leaks=1; it runs the full suite clean and was checked to
still catch a deliberately leaked allocation.
MSan: reachable from the local ladder. The image and compose service
existed but run.sh had no leg, so the only way in was to drive docker
compose by hand -- which means it was not part of the ladder in any
meaningful sense. The image also moves to clang 22, matching the diag
and analyzer lanes instead of sitting four majors behind on noble's
default. The leg documents the aarch64 shadow-mapping failure so a local
arm64 stack overflow in the grammar suites is not mistaken for a code
defect; the GitHub leg runs x86-64, which is the mapping that matters.
Off-by-default ASan checks: the diagnostic lane, and its CI twin, now run
detect_stack_use_after_return, detect_stack_use_after_scope,
detect_invalid_pointer_pairs (with the -fsanitize=pointer-compare,
pointer-subtract instrumentation it requires) and strict_string_checks.
Running ASan is not the same as running all of it, and these four cover
bug classes nothing in the matrix was looking for. They stay on the
diagnostic lane rather than the gating ones until they have a clean
history there; promoting them is a separate deliberate step.
Verified: macOS TSan 940 passed / 3 skipped / 0 races over the full
suite set; the macOS leak lane 7375 passed / 4 skipped / 0 leaks, with
LeakSanitizer confirmed armed under that exact toolchain and option set
by checking it still reports a deliberately leaked allocation.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stage 2 of the memory-diagnostics program (user decision: go directly to the
instrumented image rather than a C-only probe). MSan detects uninitialized
READS, the one memory-error class no other lane covers dynamically, and it
requires every linked library to be instrumented -- vendored C deps compile
in-tree and instrument for free; the two external links do not:
- test-infrastructure/Dockerfile.msan: pinned-base image building
libc++/libc++abi/libunwind (llvmorg-18.1.8, LLVM_USE_SANITIZER=
MemoryWithOrigins) and static zlib v1.3.1 into /opt/msan, with the
symbolizer and MSan runtime in a separate last layer so tool additions
never invalidate the ~30-min libc++ build.
- scripts/msan.sh: the canonical lane entry. ALWAYS clean-builds its
BUILD_DIR: make does not encode flags into dependencies, and a stage-1
probe's libstdc++ objects surviving into the libc++ lane produced a
convincing-looking uninitialized-value report at preprocessor.cpp:168 --
the uninstrumented .so string constructor wrote the temporary, the
instrumented move constructor read it. The clean rebuild proved it an
artifact: extraction (incl. the C++ preprocessing path) runs 272/272 with
zero reports.
- Makefile.cbm: CXX_STDLIB / CXX_STDLIB_FLAGS hooks so the lane can swap
libstdc++ for the instrumented libc++ (defaults identical; the shipping
build is byte-for-byte unaffected).
- docker-compose test-msan service: same aarch64 seccomp/setarch remedy as
the TSan service (MSan's shadow layout hits the same personality() block).
- CI test-msan job (_test.yml): buildx local-cache via the repo's existing
pinned actions/cache -- no new third-party action pins; a warm run skips
the libc++ build entirely.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel
