11 Commits
Author SHA1 Message Date
Martin Vogel 4d97d7819b fix(routes): read a ServeMux pattern only for Handle and HandleFunc
The Go 1.22 pattern split ran for every route-verb callee, so other Go
calls whose name ends in Get/Post/Put/Delete/Patch became route
registrations: st.Get("users/123") on a project method minted
__route__GET__/123 and lost its CALLS edge to Store.Get, and
c.Get("https://api.example.com/v1/x") split "https:" off as a host and
minted __route__GET__//api.example.com/v1/x, in both passes.

cbm_service_pattern_mux_route_path reads the pattern only for a callee
ending ".Handle" or ".HandleFunc", and never for an argument containing
"://". The extractor's handler capture and both passes (route_reg_path,
find_route_path_in_args, cbm_pipeline_route_reg_method) go through it,
so the two passes still mint identical Routes. Two negative cases
extend pipeline_go_servemux_method_patterns_issue2537 in both passes,
and the infrascan unit covers the gate.

Refs #2537

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-10-26 01:46:33 +01:00
Gaurav Dubey ad7899399b fix(routes): read Go 1.22 ServeMux method and host patterns
Since Go 1.22 a net/http ServeMux pattern may name a method and a host
before the path: mux.HandleFunc("POST /v1/payments", h),
"GET /v1/payments/{id}", "api.example.com/v1/x". Every route
registration gate wanted the first argument to open with a slash (the
extractor's handler capture, find_route_path_in_args in the parallel
pass, and the three gates in the sequential pass), so such a
registration minted no Route and no HANDLES edge, while the same file
written with "/v1/payments" had both.

cbm_service_pattern_mux_split reads the "[METHOD ][HOST]/[PATH]"
grammar. For Go files only, the path of such a pattern now registers
the route, the Route takes the pattern's method when the callee names
none (Handle/HandleFunc are ANY), and a host is dropped from the path.
Both passes share cbm_pipeline_mux_route_path and
cbm_pipeline_route_reg_method so their Routes stay identical. Other
languages and plain "/path" patterns take the unchanged path.

Fixes #2537

Signed-off-by: Gaurav Dubey <gauravdubey0107@gmail.com>
2026-10-21 13:23:06 +00:00
Martin Vogel d97c0ce55b fix: exclude client navigation from inferred HTTP calls
Refs #1250

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-10-13 00:33:07 +02:00
Martin Vogel 92cb0ec211 fix(pipeline): key GraphQL Routes by operation name, not operation text (#598)
When a call resolved to a GraphQL client, emit_graphql_edge named the
Route after the whole first string argument, stripping only a leading
"query " or "mutation " (the mutation strip was also one byte short, so
names kept a leading space). An anonymous `query { ... }` therefore became
one long Route, and even a named operation carried its variables and
selection set in the name ("GetUser($id: ID!) { user(id: $id) { name } }").
Every distinct operation text minted its own Route node, and names were
cut off at 256 bytes.

Add cbm_service_pattern_graphql_operation(), a small GraphQL document
scanner. It skips comments, strings, block strings, `${...}` template
interpolations and leading fragment definitions, and reports the first
operation's type and name without copying any operation text. Routes are
now keyed as follows:

  named operation              __graphql__<Name>                name <Name>
  anonymous query / mutation / __graphql__<type>__anonymous     name (anonymous <type>)
  subscription, incl. the `{ ... }` query shorthand
  no GraphQL document          __graphql__operation__anonymous  name (anonymous operation)
  (URL, bare word, no string argument)

Keys are deterministic and bounded by the GraphQL Name grammar. The
GRAPHQL_CALLS edge keeps "operation" (now the Route name) and gains
"operation_type". The old fallback to the callee name ("client.query")
is gone: a callee is not an operation.

The sequential path (pass_calls.c) emits no GRAPHQL_CALLS at all, so it
had no copy of this logic to fix.

Proof on graphql-python/gql (374a70df, Python GraphQL client, index-only):
GraphQL Routes 170 -> 16, longest QN 266 -> 34 bytes, 106 -> 0 Routes
carrying operation text. GRAPHQL_CALLS 2643 -> 1158 with the same 984
distinct callers: edges are one per (caller, operation) pair, so calls
that used to hit distinct text-keyed Routes now share their operation's
Route.

Tests (edge_types_probe): graphql_route_keyed_by_operation_name_issue598
asserts the exact GraphQL Route set (QN + name) on the parallel path.
graphql_operation_identity_parser_issue598 covers braces inside comments,
strings and interpolations, fragments before the operation, anonymous
operations with variables, name truncation and non-document arguments.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-26 10:21:30 +02:00
Martin Vogel 1a3425536b fix(graph): quality round after the v0.11.0 probe — deterministic resolution, findability, coverage, names
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.

Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.

Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).

Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.

Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.

Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).

Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-09-21 20:29:27 +02:00
Yyunozor c86f1ffafd fix(pipeline): reject non-http slash arguments
Signed-off-by: Yyunozor <yyunozor@icloud.com>
2026-08-31 20:14:01 +02:00
Alexandros Pappas d288a21ec2 fix(pipeline): classify native fetch() as HTTP_CALLS, not local shadows
Closes #856: native `fetch()` calls (Node 18+ built-in, and the identical
browser/WHATWG API — no static way to tell them apart, and no need to:
both make a real outbound HTTP request) were invisible to cross-repo
intelligence. A bare `fetch(url)` has no import and typically no local
definition, so registry resolution comes back empty and the call was
silently dropped instead of becoming an HTTP_CALLS edge.

The obvious fix — add "fetch" to the http_libraries substring table in
service_patterns.c — is wrong: that table is consulted by an early,
unconditional check in both pass_calls.c and pass_parallel.c that runs
before/regardless of whether the callee resolves to a real local
definition (by design, so `axios.get(url)` still classifies even when the
registry mis-binds bare `get` to an unrelated local method). That's safe
for "axios"/"requests" because nobody names their own function that; it
is not safe for "fetch", which collides with a plausible local identifier
(`function fetch(){}` or `const fetch = () => {}`).

Instead, cbm_service_pattern_is_global_fetch() is a new, narrow, exact-
match check consulted only in the *empty-resolution* fallback in both
files — the existing #523 path for unindexed external libraries. By the
time that branch runs, the registry has already had its chance to
resolve "fetch" to a local/imported definition; only a genuine miss
reaches the new check. This also means a member call like `repo.fetch()`
is naturally excluded (its callee_name is "repo.fetch", not the bare
"fetch" this checks for).

pass_parallel.c's empty-resolution branch calls the low-level
emit_http_async_service_edge() directly rather than going through
emit_service_edge(), which re-derives its own classification from
res->qualified_name via cbm_service_pattern_match() — a call with
"fetch" would come back CBM_SVC_NONE there and silently fall through to
a plain CALLS edge.

Three new tests in test_pipeline.c:
- bare fetch() -> HTTP_CALLS, sequential path (< 50 files)
- bare fetch() -> HTTP_CALLS, forced through the parallel resolver
  (>= 50 files)
- a local `function fetch(){}` shadowing the global -> plain CALLS to
  the local definition, zero HTTP_CALLS (the false-positive this PR
  must not introduce), bundled with a `repo.fetch()` member-call check
  in the first test

Verified end-to-end, not just by inspection: prod build clean under
-Wall -Wextra -Werror; test build clean under ASan+UBSan; full suite
5936 passed, 1 skipped (pre-existing, unrelated), 0 failed; clang-format
clean on all touched files; lint-cppcheck's pre-existing failures
(extract_defs.c, compat_regex.c) reproduced identically on unmodified
main via stash/pop, confirming they predate this change.

Refs #592.

Signed-off-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
2026-07-07 11:33:56 +08:00
Martin Vogel 1af1dd8688 perf(service_patterns): per-worker TLS cache for cbm_service_pattern_match
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.

API additions in service_patterns.h:
  void cbm_service_pattern_cache_begin(void);
  void cbm_service_pattern_cache_end(void);

Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.

Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.

Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
  baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU:    992,427 -> 885,663 ms (-10.8% vs Chunk 1,
  -5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
  AND surpasses)
- resolve_usages CPU:    11,977 -> 10,944 ms (-8.6%)
- total_loop CPU:     1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
  (within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)

Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
2026-05-21 16:28:55 +02:00
test 81385d08eb Add gRPC/GraphQL/tRPC service detection with protobuf Route extraction
Pipeline: gRPC/GraphQL/tRPC client libs in service_patterns, edge emitters
in pass_parallel, __grpc__ Route nodes from .proto service defs in
pass_route_nodes, cross-repo matchers in pass_cross_repo. Also adds MQTT,
NATS, Dapr async patterns.

Protobuf extraction: service as Class, rpc as Function with rpc_name
fallback in func_name_node. Route nodes created by line-range matching
rpc Functions to enclosing service Classes.

Tested: Online Boutique protos produce 16 gRPC Route nodes.
2026-04-15 23:53:42 +02:00
Martin Vogel b00e1f74c2 AST-based route registration + remove prescan infrastructure
Route registration detection during AST extraction: callee QN matched
against router frameworks (gin, chi, Express, Laravel, Ktor, etc.),
handler reference captured as second_arg_name, Route + HANDLES edges
created during resolution. Replaces disk-reading route discovery for
non-decorator frameworks.

Remove prescan infrastructure entirely (-350 net lines):
- cbm_prescan_t, cbm_prescan_http_site_t, cbm_prescan_route_t,
  cbm_prescan_config_ref_t structs removed from pipeline_internal.h
- prescan_http_sites, prescan_routes, prescan_config_refs,
  prescan_add_route removed from pass_parallel.c
- prescan fast-paths removed from pass_httplinks.c and pass_configlink.c
- prescan allocation/cleanup removed from pipeline.c

All post-passes now work without prescan:
- HTTP/async edges: service_patterns during resolution
- Route registration: CBM_SVC_ROUTE_REG during resolution
- Decorator routes: gbuf properties (no disk reads)
- Config file refs: disk fallback (few files)
2026-03-28 13:59:42 +01:00
Martin Vogel 80680ea367 Cross-service communication discovery + RAM-first incremental indexing
AST-based detection of HTTP calls, async dispatch (Pub/Sub, Cloud Tasks,
Kafka, SQS, etc.), and config accesses via resolved qualified names.
Route nodes as cross-service rendezvous points with infra→handler matching.
Constant propagation for module-level string assignments. YAML infrastructure
URL extraction from Cloud Scheduler configs.

RAM-first incremental pipeline: load DB into graph buffer, purge changed
file nodes, extract directly into existing buffer (resolver sees all nodes),
dump back to disk. Zero edge gap on kubernetes/django/meilisearch/neovim.

- service_patterns.c: ~170 library patterns (90 HTTP, 50 async, 30 config)
- pass_route_nodes.c: Route node creation + infra URL matching
- extract_unified.c: string constant collection + string ref classification
- extract_calls.c: first_string_arg + keyword argument extraction
- pipeline_incremental.c: RAM-first load→purge→extract→resolve→dump
- graph_buffer.c: load_from_db, delete_by_file, foreach visitors
- C++ LSP crash fix: NULL guard in cbm_type_substitute
2026-03-28 13:59:42 +01:00