mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
perf(mem): recommit purged memory on macOS so phys_footprint drops
81264814built the production allocator as mimalloc's release build (-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS charges an index worker, its phys_footprint: | peak phys_footprint | v0.11.0 | main | |---|---:|---:| | TypeScript | 1.25 GB | 1.94 GB | | dotnet/runtime | 4.5 GB | 12.7 GB | The 2026-10-09 bench missed most of this, because it sampled RSS. The difference is one compile-time branch in mimalloc's unix decommit. With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d and flagged needs_recommit. A release build leaves it "committed". When the arena hands such a range out again, it is re-accounted as a whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges every page of it to the footprint, touched or not. On macOS, take the recommit path in every build. Linux and Windows keep upstream's behaviour; there MADV_DONTNEED already drops RSS at once. Measured on Apple arm64 (macOS 26.6), mainec4fbd3avs this change, cold full indexes interleaved with main (C#: 2 runs each, in the same session as main): | corpus | footprint before | footprint after | change | wall time | |---|---:|---:|---:|---| | TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged | | kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged | | dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged | | linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s | The graph is byte-identical: TypeScript, kubernetes and django give the same nodes, edges and CALLS-set hash. dotnet/runtime is now close to v0.11.0's 4.5 GB. Measured but not taken: - the whole debug build, which costs ~5 % wall time on C#; - MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which moves the footprint; - MIMALLOC_PURGE_DELAY=0, which is already set. Vendored file changed in place (precedent1f674c80); its hash in scripts/vendored-checksums.txt is updated and security-vendored.sh passes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
1 parent
ec4fbd3a50
commit
63fb7b3db5
2 files changed
+13
-2
No files matched your search
@@ -1024,7 +1024,7 @@ ce14eb6dfd55f241095b55f70983931a2d92f7a6a622a9b360cdd70ab3b53208 vendored/mimal
|
|||||||
247a9952465eb105be03a9962e922b085ce5d52034775551a61cd8594275be73 vendored/mimalloc/src/prim/osx/prim.c
|
247a9952465eb105be03a9962e922b085ce5d52034775551a61cd8594275be73 vendored/mimalloc/src/prim/osx/prim.c
|
||||||
cc771788e5ba591efb681829a8724f8662a1f50f3db39cf5ebe7738b8ed57e44 vendored/mimalloc/src/prim/prim.c
|
cc771788e5ba591efb681829a8724f8662a1f50f3db39cf5ebe7738b8ed57e44 vendored/mimalloc/src/prim/prim.c
|
||||||
792531421c247fca0766ff7630057ec491173578ca4d1919ae34f05eb289ccf4 vendored/mimalloc/src/prim/readme.md
|
792531421c247fca0766ff7630057ec491173578ca4d1919ae34f05eb289ccf4 vendored/mimalloc/src/prim/readme.md
|
||||||
c1ae32eaac7ccf18c10058cca99723892f8bd027318590151f5011c929c7e3be vendored/mimalloc/src/prim/unix/prim.c
|
20594d38ef4fc0c641f20a8664d17ee03ce195ca21b44e036f43cdb7bcd0bd0d vendored/mimalloc/src/prim/unix/prim.c
|
||||||
bee9c655a17e1e93bf4908e0a902fd3b566628447b1446fcbfa92bca2e76ca15 vendored/mimalloc/src/prim/wasi/prim.c
|
bee9c655a17e1e93bf4908e0a902fd3b566628447b1446fcbfa92bca2e76ca15 vendored/mimalloc/src/prim/wasi/prim.c
|
||||||
f70e61a1f2fe63da2f51c581895cf4a0918a9bffb0cf8600171f297141446962 vendored/mimalloc/src/prim/windows/etw-mimalloc.wprp
|
f70e61a1f2fe63da2f51c581895cf4a0918a9bffb0cf8600171f297141446962 vendored/mimalloc/src/prim/windows/etw-mimalloc.wprp
|
||||||
4f3110ef2054c95cb275be96cb279224d3d46a728c5117bd1435425f382de778 vendored/mimalloc/src/prim/windows/etw.h
|
4f3110ef2054c95cb275be96cb279224d3d46a728c5117bd1435425f382de778 vendored/mimalloc/src/prim/windows/etw.h
|
||||||
|
|||||||
@@ -500,7 +500,18 @@ int _mi_prim_decommit(void* start, size_t size, bool* needs_recommit) {
|
|||||||
// decommit: use MADV_DONTNEED as it decreases rss immediately (unlike MADV_FREE)
|
// decommit: use MADV_DONTNEED as it decreases rss immediately (unlike MADV_FREE)
|
||||||
err = unix_madvise(start, size, MADV_DONTNEED);
|
err = unix_madvise(start, size, MADV_DONTNEED);
|
||||||
#endif
|
#endif
|
||||||
#if !MI_DEBUG && MI_SECURE<=2
|
// codebase-memory-mcp local change (2026-10-10): macOS always takes the
|
||||||
|
// recommit path that upstream reserves for MI_DEBUG / MI_SECURE>2 builds.
|
||||||
|
// In the release path a purged range stays "committed" and is re-accounted as
|
||||||
|
// a whole with MADV_FREE_REUSE when the arena hands it out again
|
||||||
|
// (arena.c, _mi_os_reuse), so macOS charges its pages to phys_footprint
|
||||||
|
// whether or not they are touched; recommitting instead charges only the
|
||||||
|
// pages that are written. Measured on the indexer (Apple arm64, macOS 26.6),
|
||||||
|
// peak phys_footprint release -> this change, wall time unchanged and the
|
||||||
|
// graph byte-identical: TypeScript 1.98 -> 1.09 GB, kubernetes 2.92 -> 1.88 GB,
|
||||||
|
// dotnet/runtime 12.7 -> 10.6 GB, linux kernel 16.0 -> 13.5 GB. Other
|
||||||
|
// platforms keep upstream's behaviour (MADV_DONTNEED drops RSS at once).
|
||||||
|
#if !MI_DEBUG && MI_SECURE<=2 && !defined(__APPLE__)
|
||||||
*needs_recommit = false;
|
*needs_recommit = false;
|
||||||
#else
|
#else
|
||||||
*needs_recommit = true;
|
*needs_recommit = true;
|
||||||
|
|||||||
Reference in new issue
Block a user