Distilled from #1245 by Andrew Hundt (a slice of 1ac3c136).
The route-registration suffix table knew .add_api_websocket_route but
not the decorator forms FastAPI and Starlette actually use:
@router.websocket("/ws")
@app.websocket_route("/events")
so a WebSocket endpoint produced no Route node on either resolver path
and was invisible to the graph. Two entries join the table, both
recorded as ANY like the existing WebSocket entry -- a socket upgrade is
not an HTTP verb. Both are needed: matching is a pure suffix compare and
.route does not match ...websocket_route, whose tail is _route.
Upstream also changed pass_calls.c, pass_parallel.c and
pipeline_internal.h. None of that is taken: main already carries the #952
sequential unresolved-callee route fallback its July base lacked, so the
table entries are the only missing piece.
RED before the fix, each form recorded on its own (the harness stops at
the first failing assert, so the two were swapped and re-run):
pipeline_python_websocket_decorators_register_routes
FAIL tests/test_pipeline.c:6163:
count_nodes_named(s, project, "/ws") (0) not >= 1 (1)
FAIL tests/test_pipeline.c:6163:
count_nodes_named(s, project, "/events") (0) not >= 1 (1)
The @app.get("/health") control above those lines passed before and
after.
GREEN after: pipeline 282 passed; pipeline parallel extraction
edge_types_probe route_canon infrascan 779 passed, 0 failed.
The test drives the sequential resolver (one file). The parallel
resolver reads the same table through cbm_service_pattern_route_method
and its suite is green; a CBM_WORKERS=4 twin of this test is a small
follow-up.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1245 by Andrew Hundt (a slice of 0370d52f).
detect_url_in_args in the parallel resolver runs for every resolved call
with no callee gating. A first argument like "/new/file.txt" clears
normalize_url_arg (it has a second slash and no whitespace) and then
reaches cbm_service_pattern_is_http_route_literal, whose filesystem
extension list knew about config, key and socket files but not about
documents or logs. So
open_file("/new/file.txt")
read_file("/docs/guide.md")
write_file("/data/app.log")
each minted a Route node and an HTTP_CALLS edge -- the graph reported a
web endpoint where the program opens a file. Five extensions join the
list: .log .md .pdf .rst .txt, alphabetical order preserved.
Two things the tests are careful about, because the obvious fixture
would not have been RED. A bare Python builtin open(...) never resolves
to an indexed target, so it never reaches the URL detector at all; the
bug shows only for calls that resolve to a project symbol, which is what
the fixture defines. And /var/... is already rejected by the filesystem
ROOT list, so the fixture paths start with segments that list does not
know, leaving the extension guard as the only thing that can reject them.
RED before the fix:
infrascan_http_route_literal_guard_rejects_document_extensions
FAIL tests/test_infrascan.c:44:
ASSERT(!(cbm_service_pattern_is_http_route_literal("/new/file.txt", "open")))
pipeline_arg_url_rejects_document_file_paths
FAIL tests/test_pipeline.c:6652:
count_nodes_named(s, project, "/new/file.txt") == 1, expected 0 == 0
The pipeline test keeps a positive control above that line -- a genuine
requests.get("/api/items") still yields its Route node and HTTP_CALLS
edge -- so the guard cannot pass by rejecting everything.
GREEN after: infrascan 4 passed; pipeline 282 passed; extraction
edge_types_probe route_canon 420 passed.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
With `import Axios from "axios"`, a common convention in TypeScript
projects, every `Axios.get/post/put/delete(url)` client call was indexed as
a route registration. The result was a `CALLS {via: route_registration}`
edge to a Route node instead of an HTTP_CALLS edge. The client's request
paths therefore posed as server routes, and no HTTP call site existed to
match a handler. microsoft/Game-Control-Puzzle-Event-Administration-Tools
has all 69 of its client calls in this shape.
Root cause: service-pattern matching is case-sensitive by design, so that
"requests" cannot match "Requests" or "myrequests". The HTTP library
table knew only the lowercase id `axios`. `Axios.get` therefore matched no
library, on either resolver. Resolution came back empty, because an
external package is never indexed. The call then fell through to the
callee-suffix fallback, which treats any `<x>.get/post/...(path)` as a route
registration.
Fix: add `Axios` next to `axios` in the HTTP library table. The table
already pairs case variants of one library (`Dio`/`dio`,
`Guzzle`/`guzzle`, `OkHttp`/`okhttp3`). The existing identifier-boundary
rule applies unchanged:
- `Axios.get` and `AxiosInstance.get` (axios's own instance type) match;
- a lowercase continuation such as `Axiosphere` does not.
Express/Koa/Fastify registrations such as `router.get(path, handler)` are
untouched.
Proof (release binary, macOS arm64,
microsoft/Game-Control-Puzzle-Event-Administration-Tools @4fe23196,
before -> after):
- route_registration CALLS 69 -> 0; all 69 were `Axios.*` calls in
client/.
- All 69 reappear as typed HTTP_CALLS from the same caller to the same
Route QN (`__route__PUT__/api/admin/users/{}`), and no typed HTTP_CALLS
edge is lost.
- HTTP_CALLS 4 -> 142. Besides the 69 typed calls, 69 are the method-less
arg_url twins that fix/http-typed-call-no-url-twin removes, and 4 are the
existing `isOnPage` arg_url edges.
- CALLS 868 -> 799.
- SEMANTICALLY_RELATED 83 -> 174, deterministic across two runs of each
binary. The semantic pass tokenizes CALLS-neighbour names, and the fake
registration edges had fed each service function's URL path into its
vector. Without them, the 69 `Axios` wrapper functions read as related.
Five sampled calls were checked against the client source, including
admin/users/service.ts:44 `Axios.put(`/api/admin/users/${participantId}/
${eventInstanceId}`)` -> `__route__PUT__/api/admin/users/{}/{}`.
On the pending ASP.NET route stack (fix/aspnet-route-case-inrepo, 8e734ee5),
together with the arg_url twin fix, the reclassified calls reach their C#
handlers:
- typed HTTP_CALLS 0 -> 69;
- DATA_FLOWS 0 -> 66, and HANDLES 94 -> 106 (+12 case_fold);
- sampled links match their controller source, for example
`Axios.put(`/api/admin/users/${participantId}`)` ->
`[Route("api/admin/users")] [HttpPut("{participantId}")] UpdateUserInfo`;
- the 3 calls left unlinked carry a query string in the URL literal.
Tests: RED 3/3 before the fix, green after, RED again on revert.
- pipeline_capitalized_axios_import_is_http_client (parallel resolver) and
its _sequential twin: `import Axios from "axios"` with Axios.get, .post
and .delete gives 0 route registrations and exactly one typed
HTTP_CALLS per call, while an Express `router.get` registration in the
same project stays a registration.
- infrascan service-pattern tables: `Axios.get`, `Axios.post` and
`AxiosInstance.get` classify as HTTP, and `geo.Axiosphere.map` stays
NONE.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
When a call resolved to a GraphQL client, emit_graphql_edge named the
Route after the whole first string argument, stripping only a leading
"query " or "mutation " (the mutation strip was also one byte short, so
names kept a leading space). An anonymous `query { ... }` therefore became
one long Route, and even a named operation carried its variables and
selection set in the name ("GetUser($id: ID!) { user(id: $id) { name } }").
Every distinct operation text minted its own Route node, and names were
cut off at 256 bytes.
Add cbm_service_pattern_graphql_operation(), a small GraphQL document
scanner. It skips comments, strings, block strings, `${...}` template
interpolations and leading fragment definitions, and reports the first
operation's type and name without copying any operation text. Routes are
now keyed as follows:
named operation __graphql__<Name> name <Name>
anonymous query / mutation / __graphql__<type>__anonymous name (anonymous <type>)
subscription, incl. the `{ ... }` query shorthand
no GraphQL document __graphql__operation__anonymous name (anonymous operation)
(URL, bare word, no string argument)
Keys are deterministic and bounded by the GraphQL Name grammar. The
GRAPHQL_CALLS edge keeps "operation" (now the Route name) and gains
"operation_type". The old fallback to the callee name ("client.query")
is gone: a callee is not an operation.
The sequential path (pass_calls.c) emits no GRAPHQL_CALLS at all, so it
had no copy of this logic to fix.
Proof on graphql-python/gql (374a70df, Python GraphQL client, index-only):
GraphQL Routes 170 -> 16, longest QN 266 -> 34 bytes, 106 -> 0 Routes
carrying operation text. GRAPHQL_CALLS 2643 -> 1158 with the same 984
distinct callers: edges are one per (caller, operation) pair, so calls
that used to hit distinct text-keyed Routes now share their operation's
Route.
Tests (edge_types_probe): graphql_route_keyed_by_operation_name_issue598
asserts the exact GraphQL Route set (QN + name) on the parallel path.
graphql_operation_identity_parser_issue598 covers braces inside comments,
strings and interpolations, fragments before the operation, anonymous
operations with variables, name truncation and non-document arguments.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #1245 by Andrew Hundt (d15071c9, the match_qn half).
match_qn was a raw strstr(qn, library_id). Short ids fired inside
unrelated words, and a service kind REPLACES the plain CALLS edge, so a
false match both invents a service edge and deletes a real call:
"gin." in plugin. -> route registration; any "/"-prefixed
first argument then minted a Route node
"nconf" in encryptionconfig -> CONFIGURES for every call into the
package
"tonic" in Monotonic -> gRPC; unparseable, so NO edge at all
"express" in expression -> route registration
"get_env" in get_envelope, get_environ -> CONFIGURES
"dio" "surf" "curl" "resty" "phin" "hyper" "treq" "rocket" in studio,
surface, curly, restyle, dolphin, hyperbolic, streq, rocketmq
The rule (qn_hit_on_boundary; every occurrence of an id is tried, not
only the first). A separator is any character that is not an ASCII
letter or digit.
before: start of string | separator | the id itself starts with a
separator ("@trpc/server") | the id starts with an uppercase
letter -- a capital opens a new CamelCase word whatever
precedes it (AsyncHttpClient, IHttpClientFactory, NSURLSession)
after: end of string | separator | the id itself ends in a separator
("gin.") | uppercase letter (GuzzleHttp, FeignClient,
KafkaProducer) | digit (urllib2, Mint.HTTP2, amqp091-go)
Rejected: a lowercase- or digit-initial id glued to a preceding letter
or digit, and any id continued by a lowercase letter. Two deliberate
differences from the upstream hunk: upstream's before-rule needed a
GuzzleHttp table entry and would have dropped IHttpClientFactory and
NSURLSession, and without the digit after-boundary urllib2 and
amqp091-go stop matching. All three match on main today.
No recall is traded for this. No boundary rule can tell "grequests"
from "myrequests", so a library whose own name glues a prefix or suffix
onto another id gets an explicit entry with the kind (and broker) it had
on main -- 47 of them, found mechanically by running main's matcher and
the new one over 173 candidate names and listing every one that went
from a kind to NONE:
HTTP grequests txrequests redaxios gaxios libcurl curlpp curlcpp
hyperlocal guzzlehttp
ASYNC aiokafka pykafka rskafka librdkafka rdkafkacpp aioamqp
amqpstorm pamqp pyamqp amqprs amqpcpp pynats jnats aiomqtt
amqtt hbmqtt umqtt mqttools emqtt rumqtt gomqtt libmosquitto
mosquittopp pubsublite gocelery
CONFIG wgetenv getenvb qgetenv dotenvx phpdotenv
ROUTE_REG apiflask honox fasthttprouter
GRAPHQL gqlparser gqlgenc aiogqlc
Left at NONE on purpose: names that are not clients of the id they
contain and were misclassified on main -- hypercorn, hyperlink,
hyperopt, openresty, gqlalchemy, celeryconfig, kafkacat, gnatsd,
pypubsub. One visible reclassification: hyperium/tonic was HTTP through
"hyper" inside "hyperium" and is now gRPC through "tonic".
RED, production reverted and the final tests kept:
infrascan_service_pattern_match_rejects_ids_inside_words
FAIL tests/test_infrascan.c:105:
svc_case_mismatches(cases) == 28, expected 0 == 0
4 passed, 1 failed
All 28 negatives mismatch on main. The 47 glued-library positives are
RED the other way round: against the boundary rule without the entries.
Each positive uses a name containing no second id, so it binds to its
own entry -- "aiokafka.AIOKafkaProducer" would have passed through
"KafkaProducer" and proven nothing.
GREEN: infrascan 5, pipeline 281, parallel 74, extraction 350,
edge_types_probe 59, route_canon 11, lang_contract 41, cross_repo 8,
registry 64, mcp 318 (4 Windows-only skips). 0 failed.
Effect on real code -- production binaries, main 92abefa3 against this
change. main against main differs by 0 edges and 0 nodes on all three
corpora, so every delta below is the change:
django CONFIGURES 76 -> 74; both calls return as CALLS
(geom.get_envelope, WSGIRequestHandler.get_environ)
typescript CONFIGURES 12,026 -> 12,024; one bogus Route "/*type*/"
gone with its CALLS and HANDLES; 3 calls return as CALLS
kubernetes CONFIGURES 7,962 -> 7,609: all 353 are "nconf" inside
encryptionconfig / authorizationconfig /
authenticationconfig, and all 353 return as CALLS.
8 bogus Routes gone -- os.ReadFile of the serviceaccount
namespace file, t.Run("/TwoWay"), "/tmp/test" -- with 26
CALLS, 2 HANDLES and 21 TESTS edges that pointed at them.
374 plain CALLS restored; 8 of them (validate.Monotonic and
friends) had NO edge at all on main. GRPC_CALLS 60 -> 60.
HTTP_CALLS 398 -> 407: nine calls main swallowed as route
registrations now reach the existing arg_url heuristic like
every other package.
88 of the 408 removed service-derived edges were read at the call site
-- all of django and typescript, 82 of 402 in kubernetes: 88 false
positives, 0 true positives lost. None of the three corpora uses a
glued-name library, so the 47 entries are covered by the unit test only.
Known limits: "_" is a separator, so optical_fiber.len still matches
"fiber." exactly as requests_get matches "requests" by design; a
lowercase id followed by a capital matches (kafkaProducer), which also
admits a dioXide-style name; a user wrapper glued without a separator
(mygetenv) is no longer classified; a glued-name library missing from
the list is NONE until someone adds it.
Co-authored-by: Andrew Hundt <ATHundt@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Nine findings from the 2026-09-16 graph-quality probe over the eleven bench corpora, each with a
test that fails without its fix and a before/after measurement on the real corpus.
Deterministic resolution (O9). Two indexes of one kernel tree differed in 5,821 IMPORTS and 632
CALLS edges. Three order/load dependencies carried it: the `#include` target was the first by-name
hit whose path ends with the include path — a bucket in worker merge order (include/linux/device.h
vs tools/virtio/linux/device.h); the registry broke suffix_match score ties by bucket position; the
LSP budget-share rule measured wall time. Include targets now prefer the File node, the least nested
path, then the lexicographic path; registry ties prefer the least nested QN, then the smaller QN; the
budget share is measured in thread CPU time like the budget itself. Two kernel indexes are now
identical (8,529,723 nodes, 1,974,940 CALLS, zero churn), as are two elasticsearch indexes.
Findability. search_graph's BM25 path honours the `label` filter and ranks an exact-name definition
above test methods that repeat the token (15/15 probe queries).
Coverage. The parse-coverage gap tree subtracts definitions the walker recovered inside an error
region (multi-line only); kernel/sched/core.c reports 476 flagged lines instead of 7,674.
Precision. Config-language callers (Makefile, CMake, YAML) no longer bind unique_name into code
(kernel Makefile -> C CALLS 195 -> 0). JS-family token names ({}, 1, (x: T) => U) are not
definitions; computed, quoted and escaped member keys stay (TypeScript 204 junk -> 0, 1,782 real
names kept). Names are cut at their first line (elasticsearch YAML 8,306 -> 0); a comment inside an
argument list is not an argument, so no Route node is named by a Java block comment; the wildcard
route `/*` is still a route. HTML/Vue/Svelte/Astro embedded member calls stay unbound.
Recall. A Python call through a self- or cls-rooted attribute keeps a specific unique_name member
target (Django +751, elasticsearch +3,288 CALLS, no new callee names).
Robustness. Edge properties never carry a split UTF-8 sequence (the 120-byte expression cut and the
JSON escaper both trim a partial code point). CLI array flags accept a JSON-array literal. Cypher
seeds a single-hop pattern from its selective end (kernel `MATCH (a)-[:CALLS]->(b) WHERE b.name =
'_printk'`: execution limit -> 7.9 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#856: native `fetch()` calls (Node 18+ built-in, and the identical
browser/WHATWG API — no static way to tell them apart, and no need to:
both make a real outbound HTTP request) were invisible to cross-repo
intelligence. A bare `fetch(url)` has no import and typically no local
definition, so registry resolution comes back empty and the call was
silently dropped instead of becoming an HTTP_CALLS edge.
The obvious fix — add "fetch" to the http_libraries substring table in
service_patterns.c — is wrong: that table is consulted by an early,
unconditional check in both pass_calls.c and pass_parallel.c that runs
before/regardless of whether the callee resolves to a real local
definition (by design, so `axios.get(url)` still classifies even when the
registry mis-binds bare `get` to an unrelated local method). That's safe
for "axios"/"requests" because nobody names their own function that; it
is not safe for "fetch", which collides with a plausible local identifier
(`function fetch(){}` or `const fetch = () => {}`).
Instead, cbm_service_pattern_is_global_fetch() is a new, narrow, exact-
match check consulted only in the *empty-resolution* fallback in both
files — the existing #523 path for unindexed external libraries. By the
time that branch runs, the registry has already had its chance to
resolve "fetch" to a local/imported definition; only a genuine miss
reaches the new check. This also means a member call like `repo.fetch()`
is naturally excluded (its callee_name is "repo.fetch", not the bare
"fetch" this checks for).
pass_parallel.c's empty-resolution branch calls the low-level
emit_http_async_service_edge() directly rather than going through
emit_service_edge(), which re-derives its own classification from
res->qualified_name via cbm_service_pattern_match() — a call with
"fetch" would come back CBM_SVC_NONE there and silently fall through to
a plain CALLS edge.
Three new tests in test_pipeline.c:
- bare fetch() -> HTTP_CALLS, sequential path (< 50 files)
- bare fetch() -> HTTP_CALLS, forced through the parallel resolver
(>= 50 files)
- a local `function fetch(){}` shadowing the global -> plain CALLS to
the local definition, zero HTTP_CALLS (the false-positive this PR
must not introduce), bundled with a `repo.fetch()` member-call check
in the first test
Verified end-to-end, not just by inspection: prod build clean under
-Wall -Wextra -Werror; test build clean under ASan+UBSan; full suite
5936 passed, 1 skipped (pre-existing, unrelated), 0 failed; clang-format
clean on all touched files; lint-cppcheck's pre-existing failures
(extract_defs.c, compat_regex.c) reproduced identically on unmodified
main via stash/pop, confirming they predate this change.
Refs #592.
Signed-off-by: Alexandros Pappas <11921291+apappas1129@users.noreply.github.com>
Imports: per-grammar parsers for hare/pascal/powershell/scheme/racket/starlark/tcl/teal/zsh + css/html/cmake/bitbake/kconfig/gn/just/nix/jsonnet/pkl/nickel/thrift/capnp/d/tablegen/crystal/fsharp/ada; node-extraction for D/PowerShell/Pascal/Julia/F#; INHERITS for those. Pipeline edges: register Variable/Field defs for WRITES/READS resolution; CONFIGURES pass consuming env_accesses; DEPENDS_ON from go.mod/requirements.txt; SQS dot-form QN. Golden updates: julia +Class, just +Function (correct new extraction).
Format drift had accumulated across recent commits (the dry-run lint enforces clang-format-20, which had not run on these commits yet). No functional changes — whitespace/layout only, verified by a clean cppcheck + clang-format + NOLINT lint-ci pass.
cbm_service_pattern_match runs once per resolved CALL edge inside
emit_service_edge — that's 7 pattern lists × ~30 patterns × strstr
= ~210 strstrs per call. On kubernetes (~600k resolved CALL edges)
the SAME resolved QN ("fmt.Errorf", "context.Context.Done",
"errors.New", etc) repeats thousands of times across files. Caching
turns the linear pattern-list scan into one hash lookup after the
first miss.
API additions in service_patterns.h:
void cbm_service_pattern_cache_begin(void);
void cbm_service_pattern_cache_end(void);
Cache is TLS (no contention) scoped to the resolve_worker's lifetime
in the parallel_resolve phase. Capacity 8192 entries per worker —
covers typical project QN diversity without rehashing.
Encoding: enum value + 1 in the void* slot (0/NULL means "miss").
Two helpers svc_enum_to_ptr / svc_ptr_to_enum hide the trick.
Benchmark on kubernetes (12 workers, 20,650 files):
- parallel_resolve wall: 98,520 -> 90,104 ms (-8.5% vs Chunk 1
baseline, -12.8% vs original 103,281 baseline)
- resolve_calls CPU: 992,427 -> 885,663 ms (-10.8% vs Chunk 1,
-5.4% vs original — Chunk 1 added overhead that Chunk 2 recovers
AND surpasses)
- resolve_usages CPU: 11,977 -> 10,944 ms (-8.6%)
- total_loop CPU: 1,005,409 -> 897,561 ms (-10.7%)
- Tests: 3616/0
- Graph: 297,082 -> 297,080 nodes (-2), 1,153,482 -> 1,156,097 edges
(within 0.2%, semantic-preserving)
- zod/serilog sanity: edges within +/-3 of baseline
- Peak RSS: 7.46 -> 7.75 GB (+290 MB for 12 × 8192-entry caches)
Cumulative trajectory: original broken state ~35 min K8s ->
1.92 min wall (-49× total). The per-file is_import_reachable cache
(Chunk 1) + the per-worker service-pattern cache (Chunk 2) together
save ~13s of resolve wall time. Next chunks target the gbuf_insert_
edge hot path and remaining strstr scans.
Pipeline: gRPC/GraphQL/tRPC client libs in service_patterns, edge emitters
in pass_parallel, __grpc__ Route nodes from .proto service defs in
pass_route_nodes, cross-repo matchers in pass_cross_repo. Also adds MQTT,
NATS, Dapr async patterns.
Protobuf extraction: service as Class, rpc as Function with rpc_name
fallback in func_name_node. Route nodes created by line-range matching
rpc Functions to enclosing service Classes.
Tested: Online Boutique protos produce 16 gRPC Route nodes.
Route registration detection now works for unresolved local variables
(app.include_router, app.mount) by matching callee name suffix when
QN resolution fails. Scans ALL arguments for URL paths — handles
prefix=, path=, route= keyword args across frameworks.
New route_reg_suffixes: include_router, mount, add_url_rule,
register_blueprint, use, register, add_route, add_api_route.
Infrastructure bindings: YAML/JSON extraction captures topic→URL
pairs from subscription configs (push_endpoint, uri). Creates
INFRA_MAPS edges linking async topic Routes to endpoint URLs.
Improved match_infra_routes: matches prefix Routes from include_router,
supports service-only matching for generic "/" handlers, correct
Cloud Run hostname parsing for new project-number format.
Martin Vogel
Yyunozor
Alexandros Pappas
Matt Van Horn
Martin Vogel
test