A construction call now resolves to __init__ only when the called
class declares that __init__ itself. An inherited __init__ is no
longer followed: class B(A) without its own __init__ keeps the B()
call on the Class B, as it was before this branch.
Following the bases bound every subclass construction to the base
constructor. On a django index (fb113765) that made Model.__init__
a 583-caller hub (every model instantiation), bound IntegerField()
to Field.__init__ and HttpResponseForbidden() to
HttpResponse.__init__, and degraded 625 constructor call sites
versus main (457 fell back to textual resolution, 168 lost their
row; CALLS -161). With own __init__ only, 6016 sites move to the
class own __init__, 5468 stay on the Class, none fall back to a
textual strategy, and total CALLS is +4 versus main.
Tests: the same-file and cross-file inherited cases now pin the
Class target, and a new same-file subclass test at the LSP and the
graph level covers B(A) without __init__ (stays on B, never
A.__init__) and C(A) with its own __init__ (binds to C.__init__,
never the class C or A.__init__). The strategy inventory comment in
the repro suite is updated to match.
Refs #1642
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Python construction call resolved to the Class node, so a user
__init__ never received a caller from ClassName(...) and the dead-code
view reported live constructors as unreferenced.
When the called name is the class itself, resolve it to the __init__
found on the class or, depth-first, its declared bases (the lookup
super().__init__() already uses), with a new lsp_constructor_init
strategy that joins the textual call site through its reason. Classes
with no user __init__ keep the Class target (lsp_constructor), as does
a builtins.* __init__. Calls through an instance (f = Foo(); f()) are
unchanged, since they dispatch to __call__.
This changes the documented Class-target contract for classes that
define __init__; the three tests that pinned it are updated.
Refs #1642
Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Replace the wall-clock ratio test pylsp_scale_cross_registry_build_linear
(py_lsp_scale suite) with pylsp_cross_registry_build_work_is_linear in
the py_lsp suite. The new test builds the same generated input (n
classes, each followed by two methods, in extraction order) and reads
the seam-build CBM_LSP_WORK counter instead of a clock, so its verdict
does not depend on host load, and it runs in PR CI, which skips the
perf suites.
4x the classes must cost at most ~4x the counted work. With the
type-QN set the build examines no registry candidates (0 at 1000 and
4000 classes); with the scanning receiver probe the work grows from
2,809,000 to 23,236,000 and the test fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings #2420 (O(1) type membership in the shared Python cross-registry
build, #2413) up to date with main a2227157.
Conflicts in internal/cbm/lsp/py_lsp.c, resolved as follows:
- Includes: main already includes src/foundation/hash_table.h (module
re-exports) and added lsp_node_iter.h. Main's lines are kept; the
branch's duplicate include is dropped.
- py_register_lsp_defs: main changed it to return bool (false when the
package scope cannot be normalized). The merged signature keeps main's
bool return and adds this branch's PyKnownTypes *known parameter.
- cbm_run_py_lsp_cross_status: main's failure handling is kept and
passes known = NULL, as the branch did for the per-file path.
- cbm_py_build_cross_registry: main added an early `return NULL` when a
def fails to register. The merged loop passes &known and frees the
type-QN set before that early return, so the set does not leak on the
failure path.
- The test seam cbm_py_lsp_test_register_defs (added on main) passes
known = NULL.
tests/test_py_lsp.c and tests/test_py_lsp_scale.c merged cleanly.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings the branch up to main a2227157. One conflict, in
py_emit_call_for: main now skips lambda replay while a class body
executes (class_body_outer_scope). The resolution keeps that guard
together with this branch's registry index:
int lambda_idx = ctx->class_body_outer_scope ? -1 : py_lookup_lambda(ctx, fname);
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pylsp_nested_and_local_classbody_calls_join_extracted_caller asserted that a class declared in a function body is NOT a def, which is the behaviour #1548 fixes. Their class-body calls still join the enclosing function; that part is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_scope_lookup_local returns the binding, not its type, so the range
tests passed a const CBMVarBinding * where a const CBMType * was
expected and the test runner did not compile
(-Werror,-Wincompatible-pointer-types). Type checks now go through
pylsp_local_type, Hidden keeps the binding so the package case still
asserts that nothing is bound, and shadow_only checks the type of its
binding.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve registry array order and full-scan fallback when optional query
storage is unavailable. Keep temporary range storage scoped to each module
operation and shared type indexes in the registry owner arena.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse a stable arena-owned function QN order for ambiguity marking and
submodule prefix lookup, preserving exact lookup policy and appended tails.
Add prefix/index failure regressions and a pipeline operation-count case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_scope_lookup_local() returns the frame binding (CBMVarBinding), not
its type, so passing it to pylsp_init_named_type() did not compile
(-Werror=incompatible-pointer-types). Add pylsp_local_type(), which
returns the binding's type field or NULL when the name is unbound, and
assert on that type as the test intended.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolve symbols declared in __init__.py under their package while retaining
the raw module identity for file and top-level caller nodes. Carry explicit
language through registry callers, select Python package imports, preserve
long exact names, and report scope allocation failures without partial output.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the arena-owned raw Module caller name so top-level resolutions
join the extractor in both pipeline drivers. Add occurrence and ownership
controls, including package modules and named lambdas.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The depth guard of the C/C++ and Python expression evaluators
(C_EVAL_DEPTH_LIMIT, PY_LSP_MAX_EVAL_DEPTH = 256) now reports a sticky
per-file failure that stops the walk and drops every LSP resolution of the
file. Both evaluators recursed once per chain link, so a 300-term sum or a
C++ fluent chain of ~130 calls counted as 256 levels of nesting: one long
expression cost its whole file every LSP call edge. A chain's length is not
nesting, and the guard must not decide graph content for it.
c_eval_expr_type and py_eval_expr_type now first walk down the chain
operands below the node (the operand the evaluator evaluates first and
derives the link's type from), then evaluate them innermost first at the
caller's depth, keeping pending links on a small arena-owned stack in the
context that nested evaluations share. When a link is evaluated, its chain
operand is already memoized, so the evaluator's own recursive call into it
is a memo hit one frame deep. Every node is still evaluated by the
unchanged inner evaluator, evaluation does not mutate scope within one
outermost call, and only operands the recursion itself evaluates are
pre-evaluated, so the types are exactly the recursive ones.
No longer counted as depth (any length):
- C/C++: binary operators other than == != < > <= >= && || (those yield
bool without evaluating an operand), member access (. and ->), calls on
a non-name callee, subscripts, conditional consequences, parentheses,
comma lists.
- Python: binary operators, attribute access, calls on obj.m(...), f()()
and (expr)(), subscripts, the else branch of a conditional, parentheses.
Still counted, unchanged guard and reporting (genuine nesting):
- C/C++: call arguments used for template argument deduction,
std::move/std::forward operands, unary, pointer and update operands,
assignment left sides, lambda return expressions, co_await, fold and
_Generic operands.
- Python: tuple/list/set elements, dictionary keys and values, the true
branch of a conditional, await operands, next()/iter()/assert_type()
arguments, called lambdas.
Proof:
- New c_lsp and py_lsp tests build chains of 300, 2,000 and 20,000 terms
(a binary + chain and a .a() receiver chain) next to an unrelated s.m().
On the previous evaluator, and with this change reverted, all four fail
at 300 terms: the depth error is recorded and the file resolves 0 calls
(C++ per-file; Python per-file and through the cross-file dispatch).
With it, every size records no error, resolves run -> S.m, and the
chain's result type resolves x.done() / the final .done().
- Real input, isolated cache, before and after:
redis 4f20cb48 (C): 36,876 CALLS, 10,699 LSP-resolved;
django fb113765 (Python): 60,123 CALLS, 25,153 LSP-resolved.
Identical CALLS edge sets on both (0 lost, 0 gained, 0 strategy
changes), 0 files with an LSP error before and after, wall time
unchanged (redis 5.5 s, django 9.1-10.2 s both ways).
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Memoize C expression evaluation, index repeated scope and registry queries,
and report incomplete walks explicitly on memo or depth failure.
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Carry declared instance-field types through extraction and result
compaction into the shared Python type registry so chained calls through
imported classes resolve their receiver types.
Refs #1277
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_emit_call_for re-walked a called lambda's body without tracking lambdas
already being walked, so a lambda that reaches itself was re-walked once per
nesting level up to the walk-depth cap, doubling with two calls per body.
mpmath's calculus/extrapolation.py grows memory at about 1.9 GB/s until it is
killed. Mark the registry entry as expanding during its re-walk and skip
re-entry.
The mpmath file now indexes in 1.1 s at about 80 MB peak RSS; the graph of
the Python 3.14 stdlib + site-packages is byte-identical.
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
The shared tier-2 registry registered Python defs one at a time without a
mid-build finalize, so every Method's receiver probe scanned all registered
types linearly: O(methods x types), single-threaded in lsp_cross_prepare.
Keep the per-def registration order and back only the existence probe with a
cbm_ht set of type QNs, as the Go and Rust builders do; the registry is
identical by construction.
On the Python 3.14 stdlib + site-packages the builder drops from 67-75 s to
0.20-0.25 s, with a byte-identical graph.
Signed-off-by: Nelson Spence <nelson@projectnavi.ai>
from m import f as g; g() produced no CALLS edge at all. Two stacked
defects: (1) py_lsp_bind_imports keyed from-style detection on the
local name matching the module-path tail - an alias never matches, so
g was bound MODULE(m.f) and the call missed; (2) even with the binding
fixed, the pass join compares the bare tail of the resolved QN (f)
against the textual callee (g) and dropped the resolution.
Fix: (1) a dotted module path whose tail differs from the local name
is an aliased binding - bind NAMED(qn) and let phase 6 upgrade it to
the registered function/class (from-import alias) or module (import
a.b as z); (2) when the identifier-callee NAMED branch resolves a name
whose textual form differs from the QN tail, emit strategy
lsp_import_alias with the textual name stashed in reason, and gate
that strategy in the join's reason list - the exact pattern already
used by lsp_dict_dispatch and lsp_func_ptr. The parallel path's hash
index misses the alias key and falls through to the same gated linear
scan.
The contract test pins all four import forms (aliased from-import,
plain from-import, single-segment module alias, dotted module alias)
so the flip from MODULE to NAMED binding cannot regress the module-
alias attribute chains.
Closes#988
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_register_instance_field mutated the shared Tier-2 registry through direct
struct writes that bypass the read_only seal: it overwrote field-type pointers
in place and reassigned the sealed type entry's field arrays to memory owned by
the per-file arena. In the fused parallel resolve path (one sealed registry
shared by all resolve workers) that is a cross-thread data race, and it leaves
the shared registry pointing into arenas that die with their file -- a
cross-file use-after-free. The count-based seal guard could not see it because
type/func counts never change.
Python extraction emits no instance fields, so resolve-time discovery is the
ONLY source of self.x / PEP-526 fields; a plain read_only gate would silently
drop real attribute-chain CALLS edges. Fix instead routes discovery into a
per-file field overlay on PyLSPContext (arena-backed, dies with the file) and
has py_lookup_field_depth consult shared-base + overlay at the exact precedence
point of the direct field scan. Mutable per-file registries keep the original
in-place write, byte-identical.
Reproduce-first, both halves pinned:
- seal_py_shared_registry_readonly_fields: RED on the bug (field count 0->1 on
a sealed registry), GREEN with the overlay.
- pylsp_fused_self_attr_chain_via_overlay: GREEN with the overlay, RED under a
simulated plain gate (self.b.work() CALLS edge lost) -- proven both ways.
Suites: c_lsp 749, py_lsp 63+45 stress/scale/bench, pipeline 216, registry 54,
scope 10, type_rep 19 -- zero failures.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_eval_expr_type evaluated a call node's attribute receiver twice (once
in the container special-case, once in the general attribute path), so an
N-link chained call cost O(2^N) evaluations — real-world ~65-link builder
chains hung indexing for hours (#710). The evaluator also recursed once
per expression nesting level with no depth guard, so a pathologically deep
expression overflowed the native stack (#720).
Three guard layers, mirroring c_eval_expr_type's design in c_lsp.c:
1. Memoization: py_eval_expr_type is now a wrapper around the renamed
py_eval_expr_type_uncached. Results are cached per node in an
open-addressing table on PyLSPContext (arena-allocated, cold per file),
keyed by node identity (TSNode.id) — NOT the start byte, since every
leftmost descendant of a chain shares it and byte-keyed entries alias
distinct nodes into silently wrong resolutions. Entries carry a scope
generation; every scope bind/restore bumps it (O(1) whole-cache flush),
so re-evaluation under changed bindings (lambda re-walks, isinstance
narrowing) behaves exactly as before. Inserts are load-bounded (<75%)
and rejected when growth fails, so a full table can never spin.
2. Depth cap: PY_LSP_MAX_EVAL_DEPTH 256 (same as C_EVAL_DEPTH_LIMIT);
past the cap the evaluator returns unknown instead of overflowing the
stack. Truncated results — including any ancestor of a truncation —
are never cached, so a depth- or budget-limited `unknown` cannot
poison post-truncation reuse.
3. Step budget: PY_EVAL_MAX_STEPS_PER_FILE 10000 (mirrors
C_EVAL_MAX_STEPS_PER_FILE); pathological files degrade to unknown
instead of stalling indexing. Cache hits don't consume budget.
Distills PR #732 (memoization; fixed the start-byte cache key and the
missing staleness handling) and PR #758 (depth cap; raised 64 -> 256 to
match the sibling LSPs, added the never-cache-truncated ordering), and
adds the step budget plus reproduce-first tests: a 40-link chain that
must resolve its final link, a heterogeneous-receiver chain that fails
under byte keying, a 30k-deep expression crash guard, and a budget
degradation guard.
Fixes#710. Fixes#720.
Co-authored-by: LA-10 <lashuha@uwaterloo.com>
Co-authored-by: Dustin Persek <dustin.persek@protonmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#228: a class/static method invoked on a cross-file imported class name
(ActionRecordX.build_from_text(...)) produced no CALLS edge, so the
method looked like dead code (in/out degree 0). The v0.7.0 py_lsp
cross-file resolver handles this; the test feeds the exact pattern through
cbm_run_py_lsp_cross and asserts the call resolves.
Closes#228.
Round 6:
CBMRegisteredFunc gains a `flags` bitfield (CBMFuncFlags) and a
`decorator_qns` NULL-terminated list. py_register_def populates them
from CBMDefinition.decorators by short-name match — so `@property`,
`@functools.cache`, `@abc.abstractmethod`, `@typing.overload`, and
`@final` all light up regardless of qualified vs bare reference. The
flags aren't consumed by resolution yet (still on roadmap: emit
distinct edge for @property attribute access vs method call); they
are recorded so future overload selection / property modeling can
use them without another schema change.
The struct extension is at-tail and existing call sites that memset
to zero stay correct; non-Python LSPs see flags=0 / decorator_qns=NULL
and behave identically.
Three new verification tests prove that generators, dataclasses, and
@property already work end-to-end through the Round 4-5 infrastructure
without further code changes:
- Generator[Foo, ...] iterables in for-loop bind loop var to Foo via
py_iterable_element_type's existing TEMPLATE handling.
- @dataclass class Point: x: Foo (class-body annotated assignment) is
picked up by Round 4's third-pass field registrar; p.x.method()
resolves through the field type.
- @property def thing(self) -> Foo: c.thing returns Foo because
attribute access on NAMED routes through py_func_return_type which
yields the getter's annotated return.
All 2895 prior tests stay green; 3 new tests pass. Total 2898 / 0.
Bench unchanged at 95% (the small bench fixture doesn't exercise
the new patterns; will expand in Round 7).
Round 5 parity push:
Subscript value typing — py_eval_expr_type's subscript branch resolves
indexing against the container's TEMPLATE type:
- TEMPLATE("dict"|"Mapping"|"defaultdict"|"OrderedDict"|...)[K] -> V
- TEMPLATE("list"|"set"|"frozenset"|"Sequence"|"Iterable"|...)[i] -> elem
- TEMPLATE("tuple", [A, B, ...])[literal_int] -> args[idx]
- Non-literal index returns UNION of all element types.
This unlocks the `self.cache[k].method()` pattern that's everywhere in
real Python. Also fixes `items[0].method()` for typed lists.
super().__init__(...) constructor edge — super().<attr> already worked
for regular methods; __init__ is special-cased because parent classes
typically don't register __init__ with a useful return type. Strategy
"lsp_super_init" with confidence 0.85.
Operator dunders — a + b looks up __add__ on a's type and uses its
return type. Same for *, -, /, //, %, **, <<, >>, &, |, ^, @. Falls
back to left-operand type when registry has no info (correct for the
common int+int / str+str / list+list cases).
3 new tests; all 2892 prior tests stay green. Bench unchanged at 95%
(fixture doesn't exercise these patterns directly — Round 6 will
expand it).
Round 4 / final parity push. Tracks self.x = expr from __init__ and
class-body annotated assignments (`x: Foo`) as registered fields on
the enclosing class type.
py_register_instance_field mutates the registry's CBMRegisteredType
field_names / field_types arrays in place. Re-registering the same
name overwrites; first-seen wins for capture order. Arena-allocated,
not a hot path.
py_lookup_field walks the alias / embedded chain with the same
CBM_LSP_MAX_LOOKUP_DEPTH cap as method lookup, so inherited fields
resolve through MRO without stack-overflow risk on pathological
hierarchies.
Field-fallback wired into py_eval_expr_type's attribute branch:
when method lookup misses, try field lookup. Returns the field's
type so downstream attribute / call resolution chains naturally.
Same fallback applied to UNION (single-match heuristic).
py_process_class now does three passes:
1. Class-body annotated assignments (`x: int = ...`) → fields.
2. __init__ / __post_init__ → self.x = expr fields.
3. All other methods.
This guarantees fields are registered before methods that reference
them, regardless of source order.
Annotated assignment binding fix: py_resolve_annotation is now called
on assignment annotations too, so `self.cache: dict[str, str] = {}`
binds with the proper TEMPLATE type instead of NAMED("dict[str, str]").
Bench result on the 136-line fixture: **95%** (36/38 calls resolved,
8.06 ms). Up from 79%. The 5% gap covers cases that need a full
constraint solver — see Final Stopping Point doc.
2 new tests; all 2890 prior tests stay green. Total 2892 / 0.
Round 3 parity push:
- match/case (PEP 634): when the case pattern is class_pattern (wrapped
in a case_pattern), narrow the match subject identifier to NAMED(class)
inside the case body. Push/pop scope per case. Capture sub-patterns
bind to UNKNOWN (field-type extraction is a v1.1 task — would require
field type info in the registry).
- await expr: identity passthrough. async def f() -> T registers with
return T, so `await f()` returning T is the resolved-call shape we
want without modeling Coroutine[Any, Any, T] explicitly.
- Expanded test_py_lsp_bench.c fixture from 65 lines to 136 lines
exercising every parity feature: dotted-module walks, dataclass +
Self chaining, multi-inheritance with super(), Optional/Union
annotations + narrowing (isinstance / is not None / walrus), generic
list[T] / dict[K, V] / defaultdict comprehensions, classmethod /
staticmethod, match/case dispatch, async/await, common stdlib
(logging.getLogger, json.loads, pathlib.Path, functools.lru_cache).
Bench result on the new fixture: 38 calls, 30 resolved, 79%, 7.50 ms.
Remaining 21% includes patterns where intermediate types lose precision
(e.g. super() constructor, dict subscript), instance attribute typing
(self.x where x isn't annotated at class scope), and decorator
return-type substitution (lru_cache).
All 2888 prior tests stay green; 2 new tests + bench expansion. Total
2890 / 0.
Round 2/3 parity push. Three connected gap fixes that share an annotation-
parser and a generic-aware iterable model.
Annotation parser upgrade:
- py_resolve_annotation now turns subscripted forms into structured types
rather than stripping. `Optional[Foo]` -> UNION(NAMED(Foo), None).
`Union[A, B]` -> UNION. `list[Foo]` / `dict[K, V]` / `Iterable[T]` ->
TEMPLATE with arg list. `X | Y` (PEP 604) handled at depth zero.
- New py_parse_type_text_qn variant: same parser without scope lookup,
used by py_register_def at registry-build time. Bare class names are
qualified to <module_qn>.<name> so registry lookups match.
- py_register_def prefers d->return_type (full subscripted text) over
d->return_types[0] when brackets are present — extract_defs's array
form strips the subscript, losing element types like Optional[Foo].
Narrowing:
- if isinstance(x, T): bind x as NAMED(T) inside the consequence body
(block-scoped via push/pop).
- if x is not None: strip None from x's UNION inside the consequence;
py_match_is_none recognizes "is not" as the single tree-sitter token
it actually is, walks through parens / walrus to find the underlying
identifier.
- Else branch unchanged — narrow's negation (not-T, including-None) is
not modeled in v1.
Walrus + truthy:
- if (x := compute()) is not None: walrus binds x in the enclosing scope
(PEP 572) so the body sees x. py_bind_walrus_in walks the cond AST
looking for named_expression / assignment_expression nodes.
Comprehension + for-loop element typing:
- New py_iterable_element_type maps TEMPLATE(list/set/Iterable/...) to
arg[0]; TEMPLATE(dict) to key arg; TEMPLATE(tuple, ...) to UNION of
args.
- Comprehensions push their own scope and bind for_in_clause loop vars
to the iterable's element type before walking body / filter clauses.
for_statement does the same.
UNION attribute access fallback (used when narrowing can't run):
- obj_type=UNION & exactly one non-None NAMED member has the attr:
emit edge with strategy=lsp_method_union, confidence 0.85.
7 new test_py_lsp.c cases. All 2880 prior tests stay green; 8 new tests
land. Bench unchanged (small fixture, no narrowing exercised there).
Parity push #1 — six gap fixes on top of the 11-phase plan:
- Parent-module bindings: `import a.b.c` now binds `a`, `a.b`, `a.b.c`
via py_bind_dotted_prefixes. Walks the dotted prefix chain on every
import, regardless of import_is_from_style. Plus an attribute-access
path on a MODULE that detects submodules: when the registry has any
function whose qn starts with `<mod>.<attr>.`, evaluating `mod.attr`
yields MODULE("<mod>.<attr>"). This makes `os.path.join(...)` and
similar resolve correctly.
- typing.cast(T, x) returns NAMED(T) (re-evaluating T as an annotation,
so generic subscripts and forward references both resolve). Detects
bare `cast(...)` and qualified `typing.cast(...)` forms.
- typing.assert_type(x, T) is a no-op at runtime; we type the result
as type-of(x).
- Forward references as quoted strings: `def f(x: "Foo")` strips quotes
and re-resolves the inner annotation. Also handles double-quoted and
single-quoted forms.
- Self return type substitution: methods declared returning `Self` /
`typing.Self` / `typing_extensions.Self` substitute to NAMED(receiver)
at call resolution time. Enables fluent / builder pattern chaining
to keep resolving methods after each step.
- Generic subscript stripping: annotations like `list[Foo]` strip to
`list` for v1; `Optional[Foo]` likewise. Container element-type
substitution still deferred (Phase 8+).
Generator additions:
- Allowlist now includes urllib, http, concurrent (top stdlib usage).
- `from X import *` re-export following at generation time. Modules
like os.path (a star-import shim of posixpath / ntpath / genericpath)
now have all forwarded definitions registered under their own QN.
Iterates to a fixed point with an 8-step ceiling. typeshed tree
expansion is bounded — only modules transitively reachable from the
allowlist are pulled in.
Stats: 137 modules, 904 classes (2797 methods), 886 free functions
(was 114 / 753 / 794). Generated file grows from 20K to ~22K lines.
6 new test_py_lsp.c cases. All 2874 prior tests stay green.
Phase 10 of Python LSP integration. Adds:
- scripts/gen-py-stdlib.py: walks a typeshed/stdlib checkout, parses
each .pyi via Python's stdlib `ast` module, and emits one C source
file populating cbm_python_stdlib_register. v1 simplifications:
- overload stacks collapse to first signature
- ParamSpec / TypeVarTuple / Concatenate skipped
- version guards (if sys.version_info >= ...) flattened to union
of branches
- per-symbol min/max version guards not yet emitted (v1.1 follow-up)
Module allowlist matches PYTHON_LSP_PLAN.md Phase 10 list — top
stdlib modules by usage in indexed Python projects, skipping
tkinter / turtle / curses / xml / email for size.
- internal/cbm/lsp/generated/python_stdlib_data.c (auto-generated,
20,021 lines): 114 modules, 753 classes (2,385 methods), 794 free
functions registered. Source pinned to typeshed commit
a7912d521e16ff63caf7a8b64b9072542be36777 (recorded in header).
Compiles cleanly under ASan + UBSan.
- lsp_all.c includes the generated file. The CBM_PYTHON_STDLIB_GENERATED
macro disables py_lsp.c's no-op stub so the real registration runs.
4 new test_py_lsp.c cases verify resolution against the registry:
os.getcwd, collections.defaultdict constructor, pathlib.Path.exists
method via typed parameter, logging.getLogger.
Known v1 limitation captured in test comments: `import os.path` only
binds the leaf `path` in scope, so `os.path.join` style chained
attribute access on the parent module name doesn't yet resolve.
Phase 10.5 will stamp parent-module bindings.
All 2869 prior tests stay green; 4 new stdlib tests pass.
Phase 9 of Python LSP integration. Implements cbm_run_py_lsp_cross and
cbm_batch_py_lsp_cross, mirroring the Go LSP pattern:
- cbm_run_py_lsp_cross builds a per-call registry from caller-supplied
CBMLSPDef[] entries (covering both file-local and cross-file
definitions), then re-parses the source via tree_sitter_python (or
uses cached_tree when supplied) and runs the standard
py_lsp_process_file pipeline.
- py_register_lsp_defs translates CBMLSPDef shapes into
CBMRegisteredType / CBMRegisteredFunc entries. Python-specific
Protocol label is recognized as an interface-like structural type.
Pipe-separated embedded_types and method_names_str are split into
arena-allocated NULL-terminated arrays. Method receivers auto-create
type entries when the type isn't otherwise registered, matching the
Go convention.
- cbm_batch_py_lsp_cross loops the per-file entry, isolating each file
in its own arena and copying string fields into the caller's arena
so they outlive the file's scratch space.
3 new test_py_lsp.c cases cover cross-file method dispatch, cross-file
inheritance (Child(Base) calling self.shared from another module), and
batch processing of two interconnected files. All 2866 prior tests stay
green; 3 new cross-file tests pass.
Phases 7-8 of Python LSP integration. Adds:
- super().method() resolution: when the call's receiver is a super() proxy
inside a class context, walk the enclosing class's embedded_types and
resolve the attribute against the first base where it exists. Edge
carries strategy="lsp_super" and confidence 0.88.
- Multi-inheritance dispatch already worked for the current registry shape
(embedded_types is a list, py_lookup_attribute iterates) — added an
explicit test to lock in the behavior.
Also fixes a Python class-body extraction gap upstream of py_lsp: the
generic extract_class_methods walker skipped any class-body child whose
node type wasn't a function_definition, which excluded everything wrapped
in a decorated_definition (i.e. every @classmethod / @staticmethod /
@property method). Patched to peek through decorated_definition for the
inner function_definition before calling resolve_method_name. This is the
root cause of the C.classmethod() / C.staticmethod() resolution miss and
the fix is independent of language-specific decorator semantics — any
language whose grammar uses decorated_definition benefits.
7 new test_py_lsp.c cases cover decorated functions, classmethod calls,
staticmethod calls, dataclass constructor + method dispatch, super() to
the right base, multi-inheritance attribute resolution on both bases, and
PEP 695-style class names without subscript handling. All 2859 prior
tests stay green; 7 new tests pass. Total: 2866 / 0.
Bundles Phases 4-6 of Python LSP integration. py_lsp.c gains a real
binder + expression evaluator + call resolver mirroring go_lsp.c's
shape:
- py_register_def populates the registry from CBMFileResult.defs.
Class/Type defs land as registered types; Method/Function defs as
registered functions with receiver_type derived from the qualified
name parent. Python's extract_defs records a class's bases as a
single "(Base, Other)" string, so we split commas at depth 0,
strip parens / generic subscripts / keyword args (metaclass=...)
to produce usable embedded_types entries.
- py_lsp_process_file walks function/class definitions, pushing scope
per function and binding self/cls to the enclosing class type
(after parameter binding so the receiver wins over an unannotated
`self` parameter).
- py_bind_parameters resolves type annotations via py_resolve_annotation:
scope -> module-qualified registry lookup -> bare NAMED fallback.
- py_eval_expr_type covers literals (int/float/str/bool/None plus list/
dict/set/tuple containers), names (via scope, then module symbol),
attribute access on MODULE / NAMED, call expressions (constructors,
module-local functions, attribute calls), comparison / boolean /
ternary expressions, and parenthesized / binary fallthrough.
- py_lookup_attribute walks the registry's embedded_types chain with
the shared CBM_LSP_MAX_LOOKUP_DEPTH cap so MRO chains terminate
cleanly.
- py_emit_resolved_call writes (caller_qn, callee_qn, strategy,
confidence) entries to result->resolved_calls. Strategies:
lsp_direct, lsp_constructor, lsp_method, lsp_module_attr,
lsp_module_attr_unresolved.
6 new test_py_lsp.c cases cover direct calls, simple method dispatch,
self.method, constructor-then-method, single-inheritance dispatch,
and the no-false-positive guarantee on UNKNOWN receivers.
All 2853 previous tests stay green; 6 new py_lsp tests pass.
Phase 3 of Python LSP integration. py_lsp_bind_imports walks the
import map populated from CBMFileResult.imports and binds each entry
into the root scope:
- `import X` / `import X as Y` -> bind to MODULE(X) (heuristic: when
module_qn does not end with `.<local_name>`).
- `from X import Y` / `from X import Y as Z` -> bind to NAMED(X.Y).
Phase 6 attribute resolution will downgrade to MODULE / class /
function as registry lookups become available.
- `from X import *` -> recorded in the import map but not scope-bound.
Phase 9 cross-file resolution will use the entry to walk re-exports.
- Relative imports (`from . import x`, `from ..pkg import y`) bind via
the same heuristic; module path is preserved so cross-file matching
can still locate the target later.
py_lsp_lookup_in_scope test hook added for unit-level scope inspection.
9 new tests in test_py_lsp.c. All 2844 prior tests stay green.
Phase 2 of Python LSP integration. Adds the py_lsp module mirroring
the go_lsp / c_lsp pattern: PyLSPContext, single-file entry
(cbm_run_py_lsp), cross-file + batch shells, stdlib registration stub.
Bodies are no-ops at this phase. Subsequent phases fill in:
Phase 3 — imports
Phase 4 — scope binding
Phase 5 — expression typing
Phase 6 — attribute resolution + method dispatch
Phase 7 — decorators
Phase 8 — class hierarchy + generics
Phase 9 — cross-file resolution + batch
Phase 10 — cbm_python_stdlib_register from typeshed
Wired into cbm_extract_file alongside the existing Go and C/C++ LSP
calls. lsp_all.c #includes py_lsp.c so the CGo umbrella picks it up
without a separate translation unit.
tests/test_py_lsp.c adds 5 smoke cases: empty source, single function,
single class, syntax error recovery, imports pass-through. The helper
shape (extract_py / find_resolved / require_resolved) matches
test_go_lsp.c so future phases can grow the suite by category.
All 2839 existing tests stay green; 5 new py_lsp tests pass.
Martin Vogel
Matt Van Horn
Nelson Spence
Martin Vogel