feat(pipeline): fuse cross-LSP into resolve worker + module-def filter

Previously cbm_pipeline_pass_lsp_cross ran as a separate sequential
pass that re-read every source file from disk and re-parsed every
tree-sitter tree on a single thread — a 50x regression on the
kubernetes Go monorepo (extract 10s, lsp_cross 9+ min when killed).

Three changes:

1. CBMFileResult gains const char *source / int source_len. The
   parallel extract worker copies source bytes into result->arena
   (per-file 100MB cap, project-wide 2GB atomic cap) so the fused
   cross-LSP step can run without disk I/O. Source pointer is
   borrowed by downstream readers; arena owns the storage.

2. Cross-LSP work moves into the parallel resolve worker. Each worker
   processes its file with the existing parallel infrastructure
   (worker pool + work stealing). Slab is reclaimed after each
   per-file cross-LSP call to bound peak slab memory. The previously
   separate lsp_cross pass entry function is kept for the sequential
   pipeline path; the parallel path no longer invokes it.

3. CBMModuleDefIndex (gopls 'package summary' pattern) — an inverted
   index module_qn -> list of def indices built once in O(D) before
   resolve_worker fires. Per file, cbm_pxc_filter_defs_for_file slices
   all_defs down to the file's own_module + imported modules only,
   typically 50-100x smaller than the global ~110k defs on kubernetes.
   Drops the per-file registry build from O(all_defs) to O(relevant).

Other plumbing: cbm_parallel_resolve signature gains all_defs,
def_count, def_modules, module_def_index params. pipeline.c builds
the index between registry_build and parallel_resolve and frees it
after. pipeline_incremental and tests/test_parallel updated to match.

Helpers in pass_lsp_cross.c exposed via the header: cbm_pxc_has_cross_lsp,
cbm_pxc_collect_all_defs, cbm_pxc_run_one(_ts), cbm_pxc_ts_modes,
cbm_pxc_build_module_def_index, cbm_pxc_filter_defs_for_file.

Kubernetes wall-time: 35 min -> 6.24 min (~5.6x) before the followup
perf commits in this series.
This commit is contained in:
Martin Vogel committed 2026-05-21 10:15:29 +02:00
1 parent 6226972f13
commit 75a0df688f
8 files changed
+503 -44

No files matched your search

+23 -5
View File
@@ -139,14 +139,32 @@ static cbm_gbuf_t *run_parallel(const char *project, const char *repo_path, cbm_
cbm_build_registry_from_cache(&ctx, files, file_count, result_cache);
/* Cross-file LSP — mirrors run_parallel_pipeline ordering in pipeline.c.
* Augments per-file resolved_calls with cross-file type-aware resolutions
* (e.g. method calls on imported classes) before parallel_resolve emits
* the call edges. */
cbm_pipeline_pass_lsp_cross(&ctx, files, file_count, result_cache);
* Build the project-wide all_defs[] precondition, then feed it into
* cbm_parallel_resolve where the fused resolve_worker invokes
* cbm_pxc_run_one(_ts) per file BEFORE materializing CALLS edges. */
char **def_modules = (char **)calloc((size_t)file_count, sizeof(char *));
int def_count = 0;
CBMLSPDef *all_defs = def_modules
? cbm_pxc_collect_all_defs(result_cache, files, file_count, ctx.project_name,
def_modules, &def_count)
: NULL;
CBMModuleDefIndex *module_def_index =
all_defs ? cbm_pxc_build_module_def_index(all_defs, def_count) : NULL;
cbm_parallel_resolve(&ctx, files, file_count, result_cache, &shared_ids, worker_count);
cbm_parallel_resolve(&ctx, files, file_count, result_cache, &shared_ids,
worker_count, all_defs, def_count, def_modules,
module_def_index);
cbm_gbuf_set_next_id(gbuf, atomic_load(&shared_ids));
cbm_pxc_free_module_def_index(module_def_index);
free(all_defs);
if (def_modules) {
for (int i = 0; i < file_count; i++) {
free(def_modules[i]);
}
free(def_modules);
}
for (int i = 0; i < file_count; i++)
if (result_cache[i])
cbm_free_result(result_cache[i]);