mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-06 17:49:47 +00:00
Martin VogelandFelix Rath
f9f44b5869
With mimalloc 3.3.2's release build, free(NULL) before the allocator's constructor dereferenced a page map that does not exist yet. glibc 2.44's newlocale() does exactly that during libstdc++ start-up, so every Linux binary died with SIGSEGV before main (reported with a reproduction in #2562). Upstream fixed it in b2226c19 (mimalloc #1341), first released in v3.4.4; v3.5.3, upstream's latest published release, contains it. Vendored from microsoft/mimalloc tag v3.5.3 (tag object deac3dcf, commit d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45): LICENSE, include/ and src/, written blob by blob from a bare clone. Before the local patches were reapplied, the staged LICENSE blob and include/ and src/ trees had the tag's exact object ids (50 files; src/arena-meta.c is gone upstream). Afterwards the only differences are two documented hunks: - src/prim/unix/prim.c: macOS always recommits on decommit, carried from63fb7b3d(the phys_footprint fix). - src/prim/osx/alloc-override-zone.c: one comment reworded, because scripts/security-vendored.sh rejects the upstream "fork()" spelling even inside a comment. No code change. The reproducible-banner patch (no __DATE__/__TIME__) is dropped: upstream removed the build clock in v3.4.5 (mimalloc #1355). The two Makefile comments that pointed at that patch now say so. The production object is built with -DMI_PROFILE=0: mimalloc 3.5.2+ compiles its allocation-sampling profiler in by default, with a sample countdown on the allocation path, and the product never uses it. vendored/mimalloc/PATCHES.md records the pin, a re-verification recipe, the check that pins each patch and the refresh procedure; THIRD_PARTY.md points at it, the SBOM version is 3.5.3 and scripts/vendored-checksums.txt is regenerated (mimalloc lines only; security-vendored.sh passes). Measured on Apple M5 Pro (macOS 26.6), full indexes, 2 interleaved reps, mainfceb25d8(3.3.2) -> this change: TypeScript peak phys_footprint 1.07/1.01 -> 1.04/1.04 GB, peak RSS 1.87/1.84 -> 1.85/1.86 GB, wall 30.6/29.1 -> 29.3/30.0 s; django footprint 0.41/0.41 -> 0.38/0.39 GB, RSS 0.59/0.59 -> 0.58/0.60 GB, wall 6.5/6.2 -> 6.5/6.5 s. The graphs are identical: full node and edge hashes, node and edge census, no dangling edges (django 68022 nodes / 322464 edges, TypeScript 306239 / 835984). The same tree without the recommit patch peaks at 2.09/2.04 GB footprint on TypeScript, so63fb7b3dstill does its job. Linux arm64 (glibc 2.39, gcc 13.3): both trees build without warnings; django peak RSS 0.67/0.68 GB on main, 0.69/0.68 GB with this change. Co-authored-by: Felix Rath <felixm.rath@gmail.com> Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
96 lines
4.3 KiB
Markdown
96 lines
4.3 KiB
Markdown
# Vendored mimalloc: source pin and local patches
|
|
|
|
## Source pin
|
|
|
|
- Upstream: <https://github.com/microsoft/mimalloc>
|
|
- Tag: `v3.5.3` (annotated tag object `deac3dcf5f0614884013d411a5c07e16bec96c9f`)
|
|
- Commit: `d4881d338125e1cb7c47ba4cfb398d6f7c0c8d45`
|
|
(tree `fbc4e8f72287d127c079ada5ede870b7d63e55f5`)
|
|
- Vendored: `LICENSE`, every file under `include/` and every file under `src/`.
|
|
Nothing else (no build files, tests or docs).
|
|
|
|
v3.5.3 is upstream's latest published release at the time of the pin
|
|
(GitHub release of 2026-09-17, on the `main3` branch).
|
|
|
|
v3.5.3 contains upstream commit `b2226c195fe884223f6d6f68ea52df476e57e5c2`
|
|
(mimalloc #1341, first released in v3.4.4): `free(NULL)` must work before the
|
|
allocator is initialized. With 3.3.2's release build, glibc 2.44's
|
|
`newlocale()` freed NULL during libstdc++ start-up, before mimalloc's
|
|
constructor, and every Linux binary died with SIGSEGV (reported with a
|
|
reproduction by Felix Rath in #2562). The preinit fixture in
|
|
`tests/test_prod_allocator_contract.sh` pins it.
|
|
|
|
## Verifying the vendored bytes
|
|
|
|
Every vendored file is byte-identical to the tag except the two hunks below.
|
|
Re-check from a fresh bare clone (nothing in it is built or run):
|
|
|
|
```sh
|
|
git clone --bare https://github.com/microsoft/mimalloc.git /tmp/mimalloc.git
|
|
git --git-dir=/tmp/mimalloc.git rev-parse 'v3.5.3^{commit}' # d4881d33...
|
|
for f in $(git --git-dir=/tmp/mimalloc.git ls-tree -r --name-only v3.5.3 -- LICENSE include src); do
|
|
[ "$(git --git-dir=/tmp/mimalloc.git rev-parse "v3.5.3:$f")" = \
|
|
"$(git hash-object "vendored/mimalloc/$f")" ] || echo "differs: $f"
|
|
done
|
|
```
|
|
|
|
The expected output is exactly the two files named below, and
|
|
`vendored/mimalloc/include` plus `vendored/mimalloc/src` must hold no file the
|
|
tag lacks.
|
|
|
|
## Local patches (reapply on every refresh)
|
|
|
|
Each patch is pinned by a check that fails when the patch is dropped.
|
|
|
|
### 1. macOS decommit always recommits: `src/prim/unix/prim.c`
|
|
|
|
In `_mi_prim_decommit`, macOS takes the recommit path that upstream reserves for
|
|
`MI_DEBUG` / `MI_SECURE>2` builds, so a purged range is not re-accounted as a
|
|
whole by `MADV_FREE_REUSE` when the arena reuses it. This cut the indexer's peak
|
|
`phys_footprint` substantially (measurements in the inline comment and in commit
|
|
63fb7b3d; re-measured on v3.5.3 when this pin was taken). Linux and Windows keep
|
|
upstream's behaviour.
|
|
|
|
Pinned by: the decommit fixture in `tests/test_prod_allocator_contract.sh`
|
|
(`needs_recommit` must be true on macOS and upstream's false elsewhere).
|
|
|
|
### 2. Comment wording: `src/prim/osx/alloc-override-zone.c`
|
|
|
|
Comment only, no code change: "crash in fork()." reads "crash while forking.".
|
|
`scripts/security-vendored.sh` rejects call spellings such as `fork(` anywhere
|
|
in vendored C, comments included, so the upstream sentence fails that gate.
|
|
(Wording first proposed in #2562.)
|
|
|
|
Pinned by: `scripts/security-vendored.sh` itself, which reports the pristine
|
|
upstream line as a blocked subprocess call.
|
|
|
|
## Build configuration
|
|
|
|
`Makefile.cbm` compiles the production object with `-DMI_PROFILE=0`, which
|
|
leaves out mimalloc's allocation-sampling profiler (on by default since 3.5.2)
|
|
and its sample countdown on the allocation path. Pinned by: the banner fixture
|
|
in `tests/test_prod_allocator_contract.sh` (a profiling build prints a
|
|
`profiling` line). The test build keeps upstream's default.
|
|
|
|
## Dropped patches
|
|
|
|
- Reproducible version banner (`src/options.c`, carried from 1f674c80 up to
|
|
3.4.4): upstream removed `__DATE__`/`__TIME__` from the banner in v3.4.5
|
|
(mimalloc #1355). The banner fixture in
|
|
`tests/test_prod_allocator_contract.sh` still asserts that the first line is
|
|
exactly `vX.Y.Z` with no build clock, so a future refresh cannot bring it
|
|
back unnoticed.
|
|
|
|
## Refresh procedure
|
|
|
|
1. Replace `LICENSE`, `include/` and `src/` with the new tag's files, deleting
|
|
files the tag no longer has, and verify the staged tree against the tag
|
|
before patching.
|
|
2. Reapply the patches above, or drop the ones upstream has absorbed (and move
|
|
them to "Dropped patches" with the upstream reference).
|
|
3. Update the pin in this file and `versionInfo` in
|
|
`scripts/ci/generate-sbom.py`, then run `scripts/security-vendored.sh --update`.
|
|
4. After `make -f Makefile.cbm cbm`, run
|
|
`bash tests/test_prod_allocator_contract.sh build/c` on Linux and macOS
|
|
(`scripts/test.sh` runs it on every CI leg), and `scripts/security-vendored.sh`.
|