perf(mem): recommit purged memory on macOS so phys_footprint drops

81264814 built the production allocator as mimalloc's release build
(-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS
charges an index worker, its phys_footprint:

| peak phys_footprint | v0.11.0 | main |
|---|---:|---:|
| TypeScript | 1.25 GB | 1.94 GB |
| dotnet/runtime | 4.5 GB | 12.7 GB |

The 2026-10-09 bench missed most of this, because it sampled RSS.

The difference is one compile-time branch in mimalloc's unix decommit.
With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d
and flagged needs_recommit. A release build leaves it "committed".
When the arena hands such a range out again, it is re-accounted as a
whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges
every page of it to the footprint, touched or not.

On macOS, take the recommit path in every build. Linux and Windows keep
upstream's behaviour; there MADV_DONTNEED already drops RSS at once.

Measured on Apple arm64 (macOS 26.6), main ec4fbd3a vs this change, cold
full indexes interleaved with main (C#: 2 runs each, in the same session
as main):

| corpus | footprint before | footprint after | change | wall time |
|---|---:|---:|---:|---|
| TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged |
| kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged |
| dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged |
| linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s |

The graph is byte-identical: TypeScript, kubernetes and django give
the same nodes, edges and CALLS-set hash.

dotnet/runtime is now close to v0.11.0's 4.5 GB.

Measured but not taken:
- the whole debug build, which costs ~5 % wall time on C#;
- MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which
  moves the footprint;
- MIMALLOC_PURGE_DELAY=0, which is already set.

Vendored file changed in place (precedent 1f674c80); its hash in
scripts/vendored-checksums.txt is updated and security-vendored.sh passes.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel committed 2026-10-10 16:38:53 +02:00
1 parent ec4fbd3a50
commit 63fb7b3db5
2 files changed
+13 -2

No files matched your search

+1 -1
View File
@@ -1024,7 +1024,7 @@ ce14eb6dfd55f241095b55f70983931a2d92f7a6a622a9b360cdd70ab3b53208 vendored/mimal
247a9952465eb105be03a9962e922b085ce5d52034775551a61cd8594275be73 vendored/mimalloc/src/prim/osx/prim.c
cc771788e5ba591efb681829a8724f8662a1f50f3db39cf5ebe7738b8ed57e44 vendored/mimalloc/src/prim/prim.c
792531421c247fca0766ff7630057ec491173578ca4d1919ae34f05eb289ccf4 vendored/mimalloc/src/prim/readme.md
c1ae32eaac7ccf18c10058cca99723892f8bd027318590151f5011c929c7e3be vendored/mimalloc/src/prim/unix/prim.c
20594d38ef4fc0c641f20a8664d17ee03ce195ca21b44e036f43cdb7bcd0bd0d vendored/mimalloc/src/prim/unix/prim.c
bee9c655a17e1e93bf4908e0a902fd3b566628447b1446fcbfa92bca2e76ca15 vendored/mimalloc/src/prim/wasi/prim.c
f70e61a1f2fe63da2f51c581895cf4a0918a9bffb0cf8600171f297141446962 vendored/mimalloc/src/prim/windows/etw-mimalloc.wprp
4f3110ef2054c95cb275be96cb279224d3d46a728c5117bd1435425f382de778 vendored/mimalloc/src/prim/windows/etw.h
+12 -1
View File
@@ -500,7 +500,18 @@ int _mi_prim_decommit(void* start, size_t size, bool* needs_recommit) {
// decommit: use MADV_DONTNEED as it decreases rss immediately (unlike MADV_FREE)
err = unix_madvise(start, size, MADV_DONTNEED);
#endif
#if !MI_DEBUG && MI_SECURE<=2
// codebase-memory-mcp local change (2026-10-10): macOS always takes the
// recommit path that upstream reserves for MI_DEBUG / MI_SECURE>2 builds.
// In the release path a purged range stays "committed" and is re-accounted as
// a whole with MADV_FREE_REUSE when the arena hands it out again
// (arena.c, _mi_os_reuse), so macOS charges its pages to phys_footprint
// whether or not they are touched; recommitting instead charges only the
// pages that are written. Measured on the indexer (Apple arm64, macOS 26.6),
// peak phys_footprint release -> this change, wall time unchanged and the
// graph byte-identical: TypeScript 1.98 -> 1.09 GB, kubernetes 2.92 -> 1.88 GB,
// dotnet/runtime 12.7 -> 10.6 GB, linux kernel 16.0 -> 13.5 GB. Other
// platforms keep upstream's behaviour (MADV_DONTNEED drops RSS at once).
#if !MI_DEBUG && MI_SECURE<=2 && !defined(__APPLE__)
*needs_recommit = false;
#else
*needs_recommit = true;