mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-11-04 10:48:20 +00:00
feat(pipeline): fuse cross-LSP into resolve worker + module-def filter
Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential pass that re-read every source file from disk and re-parsed every tree-sitter tree on a single thread — a 50x regression on the kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed). Three changes: 1. CBMFileResult gains const char *source / int source_len. The parallel extract worker copies source bytes into result->arena (per-file 100MB cap, project-wide 2GB atomic cap) so the fused cross-LSP step can run without disk I/O. Source pointer is borrowed by downstream readers; arena owns the storage. 2. Cross-LSP work moves into the parallel resolve worker. Each worker processes its file with the existing parallel infrastructure (worker pool + work stealing). Slab is reclaimed after each per-file cross-LSP call to bound peak slab memory. The previously separate lsp_cross pass entry function is kept for the sequential pipeline path; the parallel path no longer invokes it. 3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted index module_qn -> list of def indices built once in O(D) before resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices all_defs down to the file's own_module + imported modules only, typically 50-100x smaller than the global ~110k defs on kubernetes. Drops the per-file registry build from O(all_defs) to O(relevant). Other plumbing: cbm_parallel_resolve signature gains all_defs, def_count, def_modules, module_def_index params. pipeline.c builds the index between registry_build and parallel_resolve and frees it after. pipeline_incremental and tests/test_parallel updated to match. Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp, cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes, cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file. Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup perf commits in this series.
This commit is contained in:
1 parent
6226972f13
commit
75a0df688f
8 files changed
+503
-44
No files matched your search
+23
-5
@@ -139,14 +139,32 @@ static cbm_gbuf_t *run_parallel(const char *project, const char *repo_path, cbm_
|
||||
cbm_build_registry_from_cache(&ctx, files, file_count, result_cache);
|
||||
|
||||
/* Cross-file LSP — mirrors run_parallel_pipeline ordering in pipeline.c.
|
||||
* Augments per-file resolved_calls with cross-file type-aware resolutions
|
||||
* (e.g. method calls on imported classes) before parallel_resolve emits
|
||||
* the call edges. */
|
||||
cbm_pipeline_pass_lsp_cross(&ctx, files, file_count, result_cache);
|
||||
* Build the project-wide all_defs[] precondition, then feed it into
|
||||
* cbm_parallel_resolve where the fused resolve_worker invokes
|
||||
* cbm_pxc_run_one(_ts) per file BEFORE materializing CALLS edges. */
|
||||
char **def_modules = (char **)calloc((size_t)file_count, sizeof(char *));
|
||||
int def_count = 0;
|
||||
CBMLSPDef *all_defs = def_modules
|
||||
? cbm_pxc_collect_all_defs(result_cache, files, file_count, ctx.project_name,
|
||||
def_modules, &def_count)
|
||||
: NULL;
|
||||
CBMModuleDefIndex *module_def_index =
|
||||
all_defs ? cbm_pxc_build_module_def_index(all_defs, def_count) : NULL;
|
||||
|
||||
cbm_parallel_resolve(&ctx, files, file_count, result_cache, &shared_ids, worker_count);
|
||||
cbm_parallel_resolve(&ctx, files, file_count, result_cache, &shared_ids,
|
||||
worker_count, all_defs, def_count, def_modules,
|
||||
module_def_index);
|
||||
cbm_gbuf_set_next_id(gbuf, atomic_load(&shared_ids));
|
||||
|
||||
cbm_pxc_free_module_def_index(module_def_index);
|
||||
free(all_defs);
|
||||
if (def_modules) {
|
||||
for (int i = 0; i < file_count; i++) {
|
||||
free(def_modules[i]);
|
||||
}
|
||||
free(def_modules);
|
||||
}
|
||||
|
||||
for (int i = 0; i < file_count; i++)
|
||||
if (result_cache[i])
|
||||
cbm_free_result(result_cache[i]);
|
||||
|
||||
Reference in new issue
Block a user