mirror of
https://github.com/oxc-project/oxc.git
synced 2026-11-04 10:29:58 +00:00
Dunqing
74d50c1f32
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> With `--import-plugin`, `main` builds the module graph in batches of `threads × 4` entry files and waits for each batch and its dependency closure to finish before starting the next. The waiter is a pool worker that polls with `try_recv` + `yield_now`. In the profiles, the other workers spend much of the run spinning in rayon's work-stealing loop, the spinning grows toward the end of the run, and the run ends with a long tail where only a few cores are busy. The graph loop now runs on the calling thread (`rayon::in_place_scope`) and blocks on `recv`. A new batch can start while earlier ones are still open. When one open batch imports from another, union-find merges them and they finish together, so at most `3 × group_size − 1` entry files (plus their dependencies) are in flight. Without the import plugin, the loop keeps `rayon::scope` and is unchanged; scope selection and module resolution share one resolver predicate. The language server (`run_source`) and the rule test harness (`run_test_source`) choose the scope the same way, so with the import plugin they also run the graph loop on the calling thread. The speedup comes mostly from less total CPU work and a shorter tail, not from higher core utilization. samply profiles (4 threads, `-A all --import-plugin`, median of 5 runs): | Corpus | Wall | CPU | Spent in `steal` / `find_work` / `try_recv` | |---|---|---|---| | next.js | 799 → 482 ms | 1,728 → 1,272 ms | 25.3% → 0.1% of samples (438 → 2 ms) | | VS Code | 1,479 → 1,183 ms | 4,240 → 3,887 ms | 9.6% → 1.3% (402 → 51 ms) | On `main`, one worker is on-CPU for almost the whole run (next.js: 3,098 samples vs 899–1,493 on the other three workers). With this PR the four workers are even (1,124–1,405), and the top of the inverted call tree is lexer, allocator, and semantic code instead of crossbeam's deque. <img width="1290" height="358" alt="next.js: main vs this PR" src="https://github.com/user-attachments/assets/ffd00f15-4ad9-46e1-bd60-ca23c0efa696" /> <img width="1290" height="358" alt="VS Code: main vs this PR" src="https://github.com/user-attachments/assets/b917dbee-21ce-4262-b374-93af0df0beb7" /> Thread timelines from Firefox Profiler, `main` above and this PR below, same time scale. ## Results `-c empty.json --disable-nested-config --silent -A all --import-plugin`. VS Code `src` at `1306698e` (11,260 files) and next.js `packages/` at `3bccd4df` (3,404 files). Time is median (min … max). Peak RSS is the median of `wait4` max RSS. Δ time is the median of paired per-round differences. | Machine | Threads | Corpus | main | this PR | Δ time | Δ peak RSS | |---|---|---|---|---|---:|---:| | Linux x86-64, 8 vCPU | 4 | VS Code | 1,673 ms (1,566 … 2,282) | 1,403 ms (1,231 … 1,754) | −19.7% | +3.8% | | | 4 | next.js | 744 ms (649 … 949) | 444 ms (396 … 904) | −40.4% | +16.7% | | | 8 | VS Code | 960 ms (832 … 1,256) | 771 ms (661 … 900) | −18.5% | +5.6% | | | 8 | next.js | 626 ms (539 … 821) | 426 ms (374 … 509) | −30.1% | +14.1% | | Apple M4 Max (12P + 4E) | 8 | VS Code | 440 ms (427 … 494) | 359 ms (354 … 404) | −18.3% | +4.4% | | | 8 | next.js | 263 ms (248 … 282) | 191 ms (183 … 202) | −28.0% | +4.2% | | | 16 | VS Code | 378 ms (362 … 571) | 328 ms (321 … 530) | −12.6% | +8.9% | | | 16 | next.js | 238 ms (228 … 306) | 168 ms (164 … 249) | −29.1% | +2.6% | Linux: 20 rounds per row. M4 Max: 40 rounds (two passes of 20). In every 20-round set, this PR was faster in at least 17 rounds. CPU time also drops on the 8-thread Linux run: user + sys 1.73 → 1.26 s on next.js and 4.10 → 3.72 s on VS Code. Without `--import-plugin`, time and RSS are within noise on both machines: paired medians range from −6% to +3%, and this PR was faster in 16–26 of 40 rounds. On the M4 Max, lint output (`-D correctness --import-plugin -f unix`, sorted) is byte-identical to `main`. ## Memory Peak RSS is 3–17% higher with the import plugin. The largest increase is next.js on Linux; on the M4 Max the same corpus is +3% to +4%. Three things contribute: - Overlapping batches keep up to `3 × group_size − 1` entry files and their dependencies in flight, compared with one batch on `main`. - Lint jobs are spawned from the calling thread, so they go to rayon's global FIFO queue behind parses that are already queued. ASTs are linted and freed later. - The per-file arena allocators keep their capacity and the pool does not shrink, so the higher peak persists until exit. On next.js, `MIMALLOC_PURGE_DELAY=0` reduced the gap from 86 MB to 28 MB in an earlier measurement. Runs without the import plugin are unaffected. A lint-first scheduler did not reduce peak RSS on 4 cores. A cap on in-flight work is left for a follow-up. Linking `loaded_modules` stays serial on the graph thread. A parallel iterator started from a non-worker thread would queue behind pending parses and block the loop. ## Tests A unit test covers merging into a larger dependency group when `group_id` changes: it checks the new root, the accumulated counts and paths, and that the donor is cleared. Other `ModuleGroups` tests cover adding a group, path compression, and refusing to merge a finished group. The `import_plugin_multi_group` CLI fixture (9 files, `-D import/no-cycle`) matches `main`. <details> <summary>Reproduce</summary> ```sh cargo build --release -p oxlint --bin oxlint --no-default-features --features allocator echo '{}' > empty.json # Linux: taskset -c 0-3 or 0-7. macOS: --threads 8 or --threads 16. oxlint -c empty.json --disable-nested-config --silent -A all --import-plugin . ``` Run from VS Code `src` (`1306698e`) or next.js `packages/` (`3bccd4df`). Each build has its own `CARGO_TARGET_DIR`. Binaries run round-robin in rotating order with 2 warmups each. On Linux, rounds where other processes used more than 0.5 core were redone. On macOS, each round waited until other processes used less than 1.5 cores. Profiles: `samply record --rate 4000` pinned to separate cores, 1 warmup and 5 runs per binary. `main` is `31cbe378d4`. Timings were taken on an earlier revision of this branch (`be6ef780c8`), before the branch was squashed into one commit. Changes made after that revision were not re-timed: tests and refactors, a deduplicated resolver check, the same scope selection for the language server and rule tests, one fewer `Arc` clone per already-queued import, and a simpler debug check. </details> <!-- CURSOR_AGENT_PR_BODY_END -->