Pure style change.
Remove `use super::*;` statements from `disambiguate` and explicitly import all types and functions from where they come from. This makes it easier to trace dependencies.
Pure refactor.
Remove `pair!` macro in `compress` and replace it with a function. This produces identical codegen, but is IMO a bit clearer to read because you get type annotations in IDE.
Also reorder a few lines and add line breaks to make the code clearer.
Pure refactor.
`lanes` is a submodule of `pipeline`, and `pipeline/mod.rs` has the same `#![allow]` attributes, which apply to all its submodules. So the ones in `lanes.rs` have no effect. Remove them.
Tiny optimization to macros.
`Cpu::from_ident` finds the CPU feature set named by an identifier (e.g. the `Avx2` in `#[cpu_version(Avx2)]`). It compared the `Ident` with each feature set's name in turn.
Inside a proc macro, comparing a `proc_macro2::Ident` with a `&str` converts the `Ident` to a `String` every time. So that was up to 3 allocations. Convert it to a `String` once, and compare that instead.
This only affects how fast the macros run, not their output.
Define `DynCpu` separately for each architecture, so its variants can be in a different order on each. `Generic` is always first, then the SIMD feature set which the platform has, then the variants which can't exist on the platform:
- x86_64: `Generic`, `Avx2`, `Neon`.
- aarch64: `Generic`, `Neon`, `Avx2`.
- Other platforms: `Generic`, `Avx2`, `Neon`.
`Generic` is always 0, and the platform's SIMD feature set is 1. Uninhabited variants don't exist at runtime, so `DynCpu` is like a `bool`. On aarch64 with Neon enabled at compile time, and on other platforms, only one variant can exist, so `DynCpu` is zero-sized.
Each variant which can't exist on the platform is `#[expect(dead_code)]`, with no `cfg_attr` needed.
This alters codegen slightly in a few places, but those changes are entirely incidental - same instructions but in a different order, and only in code which only runs once per lex.
Pure refactor - no changes to codegen.
Now that `DynCpu` has all its variants on every platform (#28769), and all arms are compiled everywhere (#28772), `cpu_select!` macro does nothing which a plain `match` doesn't.
Replace it with `match to_dyn(cpu) { ... }` everywhere, and remove the macro.
A `match` is just Rust, so there's no macro syntax to learn, and rustfmt and Rust Analyzer handle it fully. Arms which do the same thing can also be combined with `_`, which `cpu_select!` didn't allow:
```rust
// Before
cpu_select!(cpu, {
Avx2(cpu) => bm_any_avx2(bm, nw, cpu),
Neon(_) => bm_any_generic(bm, nw),
Generic(_) => bm_any_generic(bm, nw),
})
// After
match to_dyn(cpu) {
DynCpu::Avx2(cpu) => bm_any_avx2(bm, nw, cpu),
_ => bm_any_generic(bm, nw),
}
```
This uses the `to_dyn` function, not the `Cpu::to_dyn` method. The method is an extra `#[inline(always)]` function around `to_dyn`, which rustc's MIR inliner can't inline into code which is generic over the CPU type. That changes the codegen of some callers. Those codegen changes may be incidental and have no perf impact, but impossible to know for sure without benchmarking, so using `to_dyn` is the conservative choice to avoid any accidental perf regressions.
Remove all `#[cfg(target_arch = ...)]` attributes from the lexer, except in the `cpu` and `simd` modules.
The code for every CPU feature set (AVX2, Neon and generic) is now compiled, type-checked and linted on every platform, including the tests. Rust Analyzer sees all of the code for all platforms "as one". So e.g. AVX2 code can be worked on with full IDE support on an ARM Mac, and `cargo check` / `cargo clippy` on either kind of machine catches errors in the code for the other.
Code for a feature set which the platform doesn't have still can't run, because that feature set's type is uninhabited there (see #28769). The compiler removes it.
The macros change to support this:
- `#[cpu_version]` no longer adds `#[cfg(target_arch = ...)]`. Its `#[target_feature(...)]` becomes `#[cfg_attr(target_arch = ..., target_feature(...))]`, because enabling another platform's target features is an error.
- `#[cpu_multiversion]` generates all 3 entry points (`__avx2`, `__neon` and `__generic`) on every platform, with their target features in a `cfg_attr` in the same way.
- `cpu_select!` compiles all 3 arms on every platform, instead of replacing the bodies of other platforms' arms with `unreachable!()`.
On a platform where a feature set's type is uninhabited, its `#[cpu_multiversion]` entry point starts with `unreachable!()`, before it calls the function body:
```rust
#[inline(never)]
#[cfg_attr(target_arch = "aarch64", target_feature(enable = "neon"))]
#[allow(unreachable_code)]
unsafe fn __neon(src: &[u8], n: usize, cpu: Neon) -> usize {
#[cfg(not(target_arch = "aarch64"))]
unreachable!();
__impl(src, n, cpu)
}
```
Without it, the copies of every function for that feature set (e.g. `__impl::<Neon>` on x86_64) would be part of the call graph, even though they can never run. That changes how rustc's MIR inliner treats other code, which changes the codegen of functions which only have a generic version. rustc removes code after `unreachable!()` early enough that this doesn't happen.
The call to `__impl` stays, so the entry point still uses its parameters. Without it, clippy reports lints on their types which don't apply, e.g. `ptr_arg` warning on a `&mut Vec` parameter.
With these workarounds, this PR makes no difference to generated ASM - it's a pure refactor.
Pure refactor - no change to codegen.
Preparatory step for compiling all of the lexer's code on all platforms (#28772).
`prefetch` was in `pipeline/coalesce`, and was the only code outside `simd` which used `std::arch::x86_64` intrinsics directly. Move it into `simd/prefetch.rs`, as a public free function.
Pure refactor - no change to codegen.
Preparatory step for compiling all of the lexer's code on all platforms (#28772).
Previously, the `simd/avx2` module was only compiled on x86_64, and `simd/neon` only on aarch64. So `Vec128<Avx2>` and `Vec256<Avx2>` only existed on x86_64, and `Vec128<Neon>` and `Vec256<Neon>` only on aarch64. Other platforms had no vector types at all.
Compile both modules on all platforms. On other platforms, a vector wraps a `VecData128` / `VecData256` instead of the platform's SIMD type, which has the same size and alignment:
```rust
type Vec128Inner = cfg_select! {
target_arch = "x86_64" => __m128i,
_ => VecData128,
};
```
There, the CPU type is uninhabited, so the vector can never be created.
Add a `#[cpu_impl]` attribute macro, for each `impl` block of a CPU type's vector methods. It wraps each method's body in `cfg_select!`, so the body is only compiled on that CPU's platform, and is `unreachable!()` elsewhere. Signatures don't change, so the methods exist on every platform, and code which calls them compiles everywhere.
```rust
#[cpu_impl(Avx2)]
impl Vec256<Avx2> {
pub fn movemask(self) -> u32 {
// SAFETY: `Vec256` type guarantees platform supports AVX2
unsafe { _mm256_movemask_epi8(self.inner).cast_unsigned() }
}
}
```
expands to:
```rust
#[cfg_attr(
not(target_arch = "x86_64"),
allow(dead_code, unused_variables, clippy::unused_self, clippy::needless_pass_by_value)
)]
impl Vec256<Avx2> {
pub fn movemask(self) -> u32 {
cfg_select! {
target_arch = "x86_64" => {
// SAFETY: `Vec256<Avx2>` type guarantees platform supports AVX2
unsafe { _mm256_movemask_epi8(self.inner).cast_unsigned() }
}
_ => unreachable!(),
}
}
}
```
The `simd` tests are compiled on all platforms too.
Pure refactor - no change to codegen.
Preparatory step for compiling all of the lexer's code on all platforms (#28772).
Previously, `Avx2` only existed on x86_64, and `Neon` only on aarch64, and the variants of `DynCpu` had `#[cfg]` attributes to match. `Generic` existed on all platforms, but was uninhabited on aarch64 with Neon enabled.
Make `Avx2` and `Neon` work the same way as `Generic`. All these types now exists on every platform. Each has a private field which is `()` on its own platform, and `Never` (an enum with no variants) everywhere else:
```rust
pub struct Avx2(
cfg_select! {
target_arch = "x86_64" => (),
_ => Never,
},
);
```
On platforms which can't support the CPU feature set, the type is uninhabited, so it can never be created, and the compiler removes code which handles it.
`DynCpu` has all 3 variants on every platform. Variants which can't exist on a platform are `#[expect(dead_code)]` there.
The field is private, so only code in the `cpu` module knows that a type is uninhabited. A `match` anywhere else needs an arm for every variant, including ones which can't exist. So `cpu_select!` and `#[cpu_multiversion]` add arms for them, e.g. `DynCpu::Neon(_) => unreachable!()` on x86_64.
Pure refactor - no change to codegen.
Preparatory step for compiling all of the lexer's code on all platforms (#28772).
Previously, the methods of `Vec128` and `Vec256` were inherent methods, defined separately for each CPU type in the `avx2` and `neon` modules. Code which is generic over the CPU type couldn't call them.
Move every method which has both AVX2 and Neon versions into the `Vec128Impl` and `Vec256Impl` traits (introduced in #28766) - 31 methods for `Vec128`, and 36 for `Vec256`. Each CPU type implements the traits in its own module.
`Vec128<C>` and `Vec256<C>` also have generic inherent methods with the same names, which call the trait methods. So callers don't need the traits in scope, and call sites don't change.
The structs, the traits and the generic methods are in new `simd/vec128.rs` and `simd/vec256.rs` files.
The `!`, `&`, `|` and `^` operators, and the `Operand128` / `Operand256` traits, are generic over the CPU type too. The operators move from `bit_traits.rs` into `vec128.rs` and `vec256.rs`. The `CpuType` alias, which picked the platform's CPU type with `#[cfg]`, is removed.
Methods which only one CPU has stay inherent methods of that CPU's types:
- AVX2 only: `movemask` and `Vec256::shuffle_u32s`.
- Neon only: `shl_u8`, `shr_u8`, `pairwise_add_u8` and `test_u8`.
`Vec256::shuffle_u8s` stays separate for each CPU too, because it does different things on each: 2 lookups in 16-entry tables on AVX2, 1 lookup in a 32-entry table on Neon.
Neon's `Vec256` gets the methods it didn't have yet, which work on its 2 `Vec128` halves.
Tests of the common methods move from `avx2/tests.rs` and `neon/tests.rs` into `simd/tests.rs`, and run with the CPU type which the platform has. `avx2/tests.rs` and `neon/tests.rs` keep the tests for methods which only one CPU has, or which behave differently on each.
Introduce `VecData128` and `VecData256` types, in `simd/data.rs`.
They're byte arrays aligned on 16 and 32 bytes, for lookup tables and other data which is read into SIMD vectors.
- `Vec128::read` / `Vec256::read` read one into a vector, with an aligned load.
- `Vec128::write` / `Vec256::write` write a vector into one, with an aligned store.
- `VecData128::read(cpu)` / `VecData256::read(cpu)` are the same as `Vec128::read` / `Vec256::read`, but read better for tables, e.g. `PH_A.read(cpu)`.
Convert the lookup tables which SIMD code reads into vectors: `PH_A`, `PH_B`, `PH_T`, `MERGED_LUTS`, `WORD_LUTS`, `BITS` (Neon), `QCOMPACT` and `BCOMPACT`.
The aligned read and store operations have minimal advantage over their unaligned versions. The realbenefit is ensuring that loads/stores do not touch multiple cache lines, which has a perf cost.
The change to aligned types has a positive impact for `QCOMPACT` and `BCOMPACT`, which were previously under-aligned, so reads could cross cache lines. Now that they're made up of aligned `VecData256` / `VecData128` blocks, every load is guaranteed to be within a single cache line.
`VecData128::read` and `VecData256::read` are generic over the CPU type. To support that, `CpuSimd` gets `Vec128` and `Vec256` associated types, and there are new `Vec128Impl` and `Vec256Impl` traits, which only contain `read` for now.
Both types deref to a byte array, so they can be indexed like an array, except in `const` contexts. `as_bytes_mut` and `as_u32s_mut` are for building tables in a `const` or `static`.
Part of effort to move to runtime CPU feature detection.
Most of this stack has been converting to a pattern where the crate is compiled twice for different CPU feature sets - AVX2 and generic.
In the process, a few changes occured to the ASM, resulting from different inlining decisions.
Adjust inlining to bring the codegen for AVX2 back to as close as possible to what it was before all these changes.
Add `#[cpu_version(Generic)]` attribute to every function which is not compiled for every possible CPU feature set.
This has little effect - the macro doesn't alter the function - but it does check the function receives no params which suggest it should be compiled for a specific CPU feature set (e.g. `cpu: Avx2`), and makes it clear which paths through the code are intentionally not compiled twice for AVX2 and generic.
After this PR, every function that is part of the pipeline is marked with either `#[cpu_version]` or `#[cpu_multiversion]`.
`cpu_version` macro passes through the function unchanged where the function is unsafe or has an `#[inline(always)]` attribute. It checks that the function has an appropriate param which proves that the system supports the required CPU features (e.g. `cpu: Avx2` param for a function marked `#[cpu_version(Avx2)]`.
However, that check is based on the source text only. The macro can only see the name of the type in the param, not it's actual type. Add a line to the body of the function that checks the proof - making sure the type passed in is what it appears to be e.g.:
```rust
let _: crate::cpu::Avx2 = cpu;
```
Fix how the `cpu_version` and `cpu_multiversion` macros handle attributes on the function. Apply those attributes either to the dispatcher function or the implementation function, depending on their type.
Part of effort to move to runtime CPU feature detection. Follow-on after #28681.
Speed up the proc macros that PR introduced, by reducing `syn` usage. By not parsing the bodies of functions, the compile-time cost of the macros is reduced by 2x.
Part of effort to move to runtime CPU feature detection.
The `cpu_version` and `cpu_multiversion` macros previously could not identify associated functions (methods) of types which don't receive a `self` param. Fix this by also searching the tokens of the function for `Self`.
Switch some functions back to using `Self` where previously they couldn't due to the shortcomings of the macros.
Part of effort to move to runtime CPU feature detection.
The `cpu_version` and `cpu_versions` macros had become unweildly, and we'd reached the edge of what it's possible to do with declarative macros, even using all the tt-muncher tricks. Some improvements were impractical to make.
Convert both macros to proc macros in a new `oxc_lexer_macros` crate.
The proc macros are half as many lines of code, easier to understand, and more capable.
This also allows getting rid of the weird `apply` wrapper:
```rust
// Before
#[apply(cpu_version(Avx2))]
// After
#[cpu_version(Avx2)]
```
Rename `cpu_versions` to `cpu_multiversion`, which describes it better.
Part of effort to move to runtime CPU feature detection.
Safe functions that have a `#[target_feature]` attribute require an `unsafe { ... }` block to call. Such functions also receive an `Avx2` / `Neon` token as a parameter, which proves that the function is safe to call.
Make the `cpu_version` macro wrap such functions so that they maintain their safe interface, and calling them no longer requires an `unsafe { ... }` block. The macro checks the `Avx2` / `Neon` token is present.
Introduce SIMD vector types `Vec256` and `Vec128`.
These types wrap `__m256i` / `__m128i` on x86_64, and `uint8x16_t` / `uint8x16x2_t` on aarch64.
Motivation:
1. Provide safe interface for operations which are not inherently unsafe e.g. XOR-ing 2 vectors.
2. Provide safe methods for loads from slices (which we can hopefully make more use of in the process).
3. Make it impossible to use AVX2-specific intrinsics unless on a machine which supports AVX2, by requiring an `Avx2` token for any methods which create vectors.
4. Make SIMD code less verbose and more readable, by supporting e.g. `vec1 & vec2` or `vec & 0x3F`.
5. A step towards removing all `#[cfg]` feature gates from the crate.
Migrate all the SIMD code in the crate over to using these types.
This is pure refactor. Codegen (ASM) is identical after this PR. The new types express the exact same operations, just through a different syntax.
Any function that receives a `Vec256<Avx2>` / `Vec128<Avx2>` no longer needs to also receive an `Avx2` token - receiving the vector type proves the machine supports AVX2, as it's impossible to create one without providing an `Avx2` token originally.
Some previously unsafe functions become safe, after this change.
Pure refactor.
Introduce a set of `const`s for the masks used in `MERGED_LUTS` and `WORD_LUTS`. Previously these values were hard-coded in multiple places.
Part of effort to move to runtime CPU feature detection.
Step G9 gave `is_trivia_byte` a CPU type parameter. Mark it with `#[apply(cpu_versions)]` too.
It's `#[inline]`, so it now gets entry points with `#[target_feature]`, which rustc's MIR inliner doesn't inline. LLVM inlines them everywhere instead. `is_trivia_byte` is never called out of line.
Codegen: changes in all 3 builds, in 3 functions which call `is_trivia_byte`. The plain x86_64 build's generic code goes from 36,546 instructions to 36,514, and its AVX2 copies from 24,763 to 24,733.
- `Walk::step` (Level 3): aarch64 1,470 -> 1,402 instructions, x86_64 1,313 -> 1,295, x86_64 with AVX2 1,303 -> 1,287. It now calls `Walk::push` out of line at one call site, where it inlined it at all of them before. - `type_args_at`: aarch64 1,730 -> 1,702, x86_64 1,724 -> 1,710 (generic copy), x86_64 with AVX2 1,705 -> 1,691 (generic copy). It calls the same functions as before. - The AVX2 copy of `gt_run_split` in the x86_64 build, which inlines the AVX2 copy of `type_args_at`: 2,379 -> 2,349.
Part of effort to move to runtime CPU feature detection.
The same as step G8, for the functions in `token.rs` which the pipeline calls: `TokenKind::from_u8`, `TokenKind::from_u8_unchecked`, `StringSpan::new`, `StringSpan::with_cooked_invalid` and `is_trivia_byte`. They're private to the crate, so adding a parameter doesn't change the public API.
- All 5 take `cpu: impl Cpu`, or `_cpu` where nothing they call uses it. - `TokenKind::is_trivia` and `debug_assert_kind_bytes` are single-copy code, so they start with `let cpu = BASELINE;`. So does `build_spans_generic`. `build_spans_avx2` and `Lanes::push_string_raw` rename `_cpu` to `cpu`. Tests pass `BASELINE`. - `is_trivia_byte` isn't marked yet, because marking it changes codegen.
`from_u8` and `from_u8_unchecked` have no `self`, so `cpu_versions!` puts their entry points inside them, as for a free function, and `Self` can't be used there. They name `TokenKind` instead. The macro's docs say so.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
`unicode_ws_len` is compiled for each CPU feature set, as part of `unicode_ws_len_at` and `unicode_ws_len_at_unchecked`. Mark it with `#[apply(cpu_versions)]`, and give it a `_cpu: impl Cpu` parameter, which both callers pass.
Codegen: one change, in both copies of `unicode_ws_len_at`, in both x86_64 builds. aarch64 is unchanged.
The instructions are the same, but LLVM merges the function's `pop rbp; ret` tails differently. So the jump table's default target and 3 branches go to a different, identical `ret` block.
rustc's MIR inliner doesn't inline `unicode_ws_len` into `unicode_ws_len_at`, with or without the parameter, so LLVM inlines it. With the parameter, LLVM gets one `unicode_ws_len` for each CPU type, instead of one shared by both.
Part of effort to move to runtime CPU feature detection.
Mark every function which is compiled for each CPU feature set with `#[apply(cpu_versions)]`, whatever its inline attribute. 78 functions in all. Most are `#[inline(always)]`, which the macro checks and then outputs unchanged. They include the `cpu_select!` dispatchers, helpers which both an AVX2 version and a generic version call (e.g. `glue_hyphen_at`, `lic_verify_at` and `len_mask_ptr`), `token_view`, and the Level 3 functions which versioned code calls (`Walks::restart`, `Walk::reset` and `Walks::run_rules`).
43 of them had no CPU type parameter. They now take `cpu: impl Cpu`, or `_cpu` where nothing they call uses it. Their callers pass `cpu`:
- Callers which had `_cpu` rename it to `cpu`. - Generic versions, single-copy code (Level 3 and cold functions), and the `const` code which builds tables start with `let cpu = BASELINE;`. - Tests pass `BASELINE`.
`unicode_ws_len` isn't marked yet, because giving it a CPU type parameter changes codegen.
The docs of `cpu_versions!` no longer say it's only for functions which aren't always inlined.
Codegen: no change in any build. `Walks::restart` and `Walk::reset` get entry points, which LLVM inlines as before. Panic line numbers in `disambiguate/common/bits.rs` change.
Part of effort to move to runtime CPU feature detection.
Mark the generic versions of functions which `cpu_select!` picks between, and the helpers which only they use (`load64`, `eqm`, `lic_first8` and `pk`), with `#[apply(cpu_version(Generic))]`. 18 functions in all.
Generic versions have no CPU type parameter. `cpu_select!` calls them from its `Generic` arm, and from its `Neon` arm where there's no Neon version. Either way, the CPU type is `BASELINE`'s. So where a generic version needs a CPU type, to call other code, it uses `BASELINE`.
`cook_short_generic` and `lanes_post_generic` took `cpu: impl Cpu`, and had `#[apply(cpu_versions)]`. They lose both, and start with `let cpu = BASELINE;`. Their dispatchers' `Neon` and `Generic` arms no longer pass `cpu`.
`cls_table` isn't marked. It's a `const fn` which only builds `static` tables at compile time.
Codegen: no change in any build. Dropping `cpu_versions` from `cook_short_generic` and `lanes_post_generic` didn't let rustc's MIR inliner inline them. Panic line numbers in `lanes.rs` and `scan/generic.rs` change.
Part of effort to move to runtime CPU feature detection.
`get_pair_luts` had `#[apply(cpu_versions)]`, which marks a function compiled for each CPU feature set. But only `compress_blocks_avx2` calls it, so it's an AVX2-only helper. Mark it with `#[apply(cpu_version(Avx2))]`, and change its parameter from `impl Cpu` to `Avx2`.
Codegen: one change, in `compress_blocks_avx2` in the plain x86_64 build (107 -> 144 instructions). Other builds are unchanged.
In that build, `DynCpu::detect` never returns `Avx2` yet (detection is still at compile time), so `PAIR_LUTS` is never initialized. LLVM used that to delete the address arithmetic of the lookup table loads, because `get().unwrap_unchecked()` on a `OnceLock` which is never set is UB. The function is unreachable in that build, so this never ran.
With `get_pair_luts` inlined differently, LLVM no longer does that. `compress_blocks_avx2` is now instruction-for-instruction identical to its version in the build with AVX2 enabled, which is what runtime detection will produce.
Part of effort to move to runtime CPU feature detection.
Mark the AVX2 and Neon versions of functions which `cpu_select!` picks between, and the helpers which only they use, with `#[apply(cpu_version(Avx2))]` or `#[apply(cpu_version(Neon))]`. 31 functions in all. The macro adds the `#[cfg(target_arch = ...)]` and `#[target_feature(...)]` attributes, so the hand-written ones are removed (14 `#[target_feature]`s and 7 `#[cfg]`s).
- `array_map` and `lic_first` had no CPU type parameter. They now take `_cpu: Avx2`, and their callers pass it. - `init_pair_luts_avx2` had no `#[target_feature]`, and now gets one from the macro. So its callers call it in an `unsafe` block, and it calls `PairLuts::new` without one. `PairLuts::new`'s `# SAFETY` section, which explained that `unsafe` block, is removed. - Remove the temporary `#[expect(unused_*)]`s which step G5 added.
`qcompact` and `bcompact` aren't marked. They're `const fn`s which only build `static` tables at compile time. Nor is `uninit_array_mut`, a function nested inside `PairLuts::new`. `get_pair_luts` keeps `cpu_versions` for now, because changing it changes codegen.
Codegen: no change in any build. Panic line numbers in `lanes.rs` change.
Part of effort to move to runtime CPU feature detection.
Add `cpu_version!`, which marks a function as the version of some code for one CPU feature set. Applied with `#[apply(cpu_version(Avx2))]`, `#[apply(cpu_version(Neon))]` or `#[apply(cpu_version(Generic))]`.
- `Avx2`: The last parameter must be `cpu: Avx2`. Adds `#[cfg(target_arch = "x86_64")]`, and `#[target_feature(enable = "avx2,bmi2,popcnt")]` unless the function is `#[inline(always)]`. - `Neon`: The same, with `cpu: Neon`, `aarch64` and `neon`. - `Generic`: No parameter can have a CPU type. A generic version only runs where the CPU feature set is `BASELINE`'s, so it uses `BASELINE` where it needs one.
The function is output from its original tokens, so it compiles and is linted exactly as written.
The macro isn't used outside tests yet. The next steps apply it, and remove the `#[expect(unused_*)]`s.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
`cpu_versions!` used to reject `#[inline(always)]` functions, because `#[inline(always)]` can't be combined with `#[target_feature]` on the entry points. Now it checks them like any other function (signature, generics, parameters, `cpu: impl Cpu`), then outputs them unchanged, with no entry points.
An `#[inline(always)]` function is compiled for each CPU feature set as part of its callers' entry points. So later steps can mark it with `cpu_versions` too, as code which is compiled for each feature set.
The function is output from its original tokens, kept in a new `item` slot. So it compiles, and is linted, exactly as if the macro weren't there (e.g. `clippy::inline_always` still fires, so an `#[expect]` of it is still fulfilled).
Tests cover an `#[inline(always)]` function and method.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
`cpu_versions!` accepts generic parameters before the const ones (e.g. `<T, const N: usize>`), so that a later step can apply it to `generic functions.`
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
`cpu_versions!` accepts lifetime parameters before the const ones (e.g. `<'a, const N: usize>`), so that a later step can apply it to `token_view`. Lifetimes can't have bounds.
Every generated function declares the lifetimes. Calls between them don't pass them: rustc infers them, and a lifetime which appears in the parameters' types can't be passed explicitly. The turbofish is left out when there are no const parameters.
Tests cover a function with only a lifetime, and a function and a method with a lifetime and a const parameter.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
Move `pipeline/cpu/` to `cpu/`, so that code outside `pipeline`, like `token.rs`, can use the CPU types in later steps. The macros' paths change from `crate::pipeline::cpu` to `crate::cpu`. In imports, `cpu` joins the `use crate::{..}` group of crate-root modules.
The module is no longer under `pipeline`'s lint policy, which allows `clippy::pedantic`, `clippy::nursery` and `unsafe_op_in_unsafe_fn`:
- `redundant_pub_crate` gets a module-level `#[expect]`. The items keep `pub(super)`, which states their visibility explicitly. - Both `to_dyn`s expect `clippy::inline_always`, as `StringSpan` does. - The test `read_byte` expects `unsafe_op_in_unsafe_fn`. `cpu_versions!` doesn't wrap the calls between its entry points in `unsafe` blocks. `pipeline`'s lint policy hid that. A later step fixes it.
Codegen: no change in any build. Only symbol names change, because they include the module path of the CPU types.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift from an earlier step in this stack.
Before the migration, `Tokens::base_kind` was inlined everywhere. Since C7, Level 3's `Walk::step` called it out of line, from inside the `Walk::asi` it inlines.
It only reads the kind lane, and needs no CPU type. So it gets `#[inline(always)]` instead of `cpu_versions!`, and loses its `cpu` parameter, as in step B12. Its callers stop passing `cpu`. Level 3's `scan::after` and `Walk::continues_after_type_break` no longer need `let cpu = BASELINE;`.
Codegen: `base_kind` is inlined everywhere again, in all builds. aarch64 grows from 35,333 instructions to 35,356. In the x86_64 build, the generic code grows from 36,506 instructions to 36,546, and the AVX2 copies shrink by 54.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift in Level 3 of `disambiguate` from an earlier step in this stack.
Before the migration, `Walk::asi`, `Walk::continues_after_type_break`, `Walk::end_region_for`, `Walk::pop` and `Walk::pop_concise` were inlined everywhere. Since C7, `Walk::step` called them out of line. LLVM now rates those call sites as cold, so it inlines nothing there which costs more than 45.
They get `#[inline(always)]`. They're Level 3 code, compiled only once, so they take no CPU type.
Codegen: they're all inlined everywhere again, in all builds. `Walk::step` still calls `Tokens::base_kind` out of line, from inside the inlined `asi`, and so does `Tokens::word_kw` on aarch64. aarch64 shrinks from 35,445 instructions to 35,333. In the x86_64 build, the generic code shrinks from 36,586 instructions to 36,506, and the AVX2 copies are unchanged.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift from on earlier step in this stack.
Before the migration, `Tokens::next_start` was inlined everywhere. Since `cpu_versions!` was applied in C7, Level 3's `Walk::step` called its generic copy out of line.
It only wraps the `#[inline(always)]` `bits::next1`, and needs no CPU type. So it gets `#[inline(always)]` instead of `cpu_versions!`, and loses its `cpu` parameter, as in step B12. Its callers stop passing `cpu`. `Tokens::next_sig` and `Tokens::run_len` now pass `cpu` to nothing, so they rename it to `_cpu`. `Walk::advance` no longer needs `let cpu = BASELINE;`.
Codegen: `next_start` is inlined everywhere again, in all builds. aarch64 grows from 35,428 instructions to 35,445. In the x86_64 build, the generic code grows from 36,553 instructions to 36,586, and the AVX2 copies grow by 4.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift from on earlier step in this stack.
Before the migration, `push_string`, `push_template` and `push_atom` were inlined into `carve`, `carve_jsx` and `compress`, and `cook` was called out of line. Since `cpu_versions!` was applied, it was the other way round. Copies of the `push_*` methods were called out of line, with `cook` inlined into them.
The `push_*` methods are small wrappers around `cook`, so they get `#[inline(always)]` instead of `cpu_versions!`. They keep `cpu`, because `cook` is versioned.
Codegen: restores the original shape in all builds. The `push_*` methods are inlined again, and `cook` is called out of line again. aarch64 grows from 35,237 instructions to 35,428 (35,511 before the migration). In the x86_64 build, the generic code grows from 36,450 instructions to 36,553 (36,523 before the migration), and the AVX2 copies grow by 92.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift from on earlier step in this stack.
Before the migration, `line_break_in` was inlined everywhere, and the `line_terminator_after` it wraps was called out of line. Since `cpu_versions!` was applied in C11, it was the other way round. Copies of `line_break_in` were called out of line from Level 2 (`gt_run_split`, `type_args_at`), with `line_terminator_after` inlined into them.
`line_break_in` only wraps `line_terminator_after`, so it gets `#[inline(always)]` instead of `cpu_versions!`. It keeps `cpu`, because `line_terminator_after` is versioned. Its doc comment moves above its attribute, where it was before `cpu_versions!` needed to be the first attribute.
Codegen: restores the original shape in all builds. `line_break_in` is inlined again, and `line_terminator_after` is called out of line again.
Part of effort to move to runtime CPU feature detection. Fixes codegen drift from an earlier step in this stack.
Before the migration, both were inlined everywhere, and the functions they wrap, `walk::ident_is` and `Tokens::word_kw`, were called out of line. Since `cpu_versions!` was applied in C7, it was the other way round. Their copies were called out of line from Level 2 (`gt_run_split`, `type_args_at`) and from `Tokens::prev_token`, with the wrapped functions inlined into them.
They only wrap other functions, so they get `#[inline(always)]` instead of `cpu_versions!`. They keep `cpu`, because the functions they call are versioned.
Codegen: restores the original shape in all builds. `Tokens::ident_is` and `Tokens::ident_kw` are inlined again, and `walk::ident_is` is called out of line again. aarch64 grows from 34,534 instructions to 35,241 (35,511 before the migration). In the x86_64 build, the AVX2 copies grow by 123 instructions, and the generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection. Follow-up after auditing codegen after work so far in this stack.
`Walks::run_rules` is Level 3 code, but `gt_run_split` calls it from Level 2. Outside tests, it only returns `true`, and that must fold into `gt_run_split`. So it gets `#[inline(always)]` instead of `cpu_versions!` or `#[inline(never)]`. Its test version too, so both versions are the same shape.
Codegen: no change in any build. It was already inlined everywhere.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `get_pair_luts`. It doesn't use `cpu` yet. Its only caller `compress_blocks_avx2` passes its `Avx2`, so it renames `_cpu` to `cpu`.
Codegen: no change in any build. `get_pair_luts` was already inlined into `compress_blocks_avx2`, and still is.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Lexer::ensure`. It doesn't use `cpu` yet. `lex_raw_impl` passes `cpu`. `Lexer::lex` also calls it, outside the runtime switch, so it uses `BASELINE`, which adds no runtime dispatch.
Codegen: aarch64 is unchanged, with `ensure` out of line as before. On x86_64, the AVX2 copy of `ensure` has only one caller, so it's now inlined into the AVX2 copy of `lex_raw_impl`, which grows by about 410 instructions. It runs once per lex. `Lexer::lex` and the generic copy of `lex_raw_impl` call the generic copy out of line, as before. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `utf8_seq_check`, keeping its `#[inline]`. It doesn't use `cpu` yet. Its only caller `misc_pre_impl` passes `cpu`.
Codegen: no change in any build. `utf8_seq_check` was already inlined into each copy of `misc_pre`, and still is.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `write_sentinels`. It doesn't use `cpu` yet. Its callers `lex_raw_impl` and `compress` pass `cpu`.
Codegen: no change in any build. `write_sentinels` was already inlined into both callers, and still is.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Lanes::clear`. It doesn't use `cpu` yet. Its only caller `lex_raw_impl` passes `cpu`.
Codegen: no change in any build. `Lanes::clear` was already inlined into `lex_raw_impl`, and still is.
Part of effort to move to runtime CPU feature detection.
`token_view` is reachable from `lex_raw_impl`, but `cpu_versions!` can't take its explicit lifetime parameter. A separate copy for each CPU wouldn't gain anything, because its body only builds a `Tokens` from slices. It's already inlined at all 7 call sites in every build, so `#[inline(always)]` keeps the status quo.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
`load256`, `veq` and `mm` take `_cpu: Avx2`. Their callers `classify_avx2`, `find_avx2`, `bracket_bits_avx2`, `any_of`, `scan_block_comment_avx2`, `scan_line_comment_avx2` and `jsx_glue_hyphens_avx2` pass their `Avx2`. The last 4 rename `_cpu` to `cpu`.
The `Avx2` type now guarantees that the CPU supports AVX2, so `veq` and `mm` are safe functions, like `veq` in `classify/neon.rs`. `load256` stays `unsafe`, because of its pointer. `bracket_bits_avx2` no longer needs an `unsafe` block around `mm`.
Codegen: no change in any build.
Part of effort to move to runtime CPU feature detection.
Replace the cfg'd versions of `jsx_glue_hyphens` with an `#[inline(always)]` dispatcher which takes `cpu` and uses `cpu_select!`. Both versions stay `#[inline(always)]`, and `jsx_glue_hyphens_avx2` takes `_cpu: Avx2`. Its only caller `carve_jsx` passes `cpu`.
Codegen: aarch64 is unchanged. With AVX2 flags, the AVX2 copy of `carve_jsx` is unchanged, and its generic copy now uses the generic version, as intended. In the x86_64 build, the AVX2 copy now uses the AVX2 version. It now has the same instructions as in the build with AVX2 flags, as do the AVX2 copies of `carve` and `coalesce`. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Replace the cfg'd versions of `span_has_bs` and `span_has_bs_or_cr` in `lanes.rs` with `#[inline(always)]` dispatchers which take `cpu` and use `cpu_select!`. The AVX2 versions get `#[target_feature]` and take `_cpu: Avx2`. They no longer need `unsafe` blocks around the intrinsics which don't access memory. Their only caller `Lanes::cook` passes `cpu`.
Codegen: aarch64 is unchanged. With AVX2 flags, the AVX2 copies are unchanged, and the generic copies of `push_string`, `push_template` and `push_atom` now use the `memchr` versions, as intended. In the x86_64 build, their AVX2 copies now inline the AVX2 versions instead of calling `memchr`. They now have the same instructions as in the build with AVX2 flags. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Replace the 3 cfg'd versions of `keywords::text_matches` with an `#[inline(always)]` dispatcher which takes `cpu` and uses `cpu_select!`. All 3 versions stay `#[inline(always)]`. `text_matches_avx2` takes `_cpu: Avx2` and `text_matches_neon` takes `_cpu: Neon`.
`kw_match` takes `cpu`. Its callers `kw_verify_batch`, `kw_match_word` and `kw_match_at` pass theirs. Tests pass `BASELINE`.
Codegen: aarch64 is unchanged. With AVX2 flags, the AVX2 copies are unchanged. The generic copies of `kw_match`'s callers, and the Level 3 functions which reach it, now use the generic version, as intended. In the x86_64 build, the AVX2 copies now use the SSE2 version. Their inlining now matches the build with AVX2 flags, with `Tokens::word_kw` inlined and `Tokens::ident_kw` out of line. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Replace the cfg'd aliases in `scan/mod.rs` with `#[inline(always)]` dispatchers `scan_block_comment` and `scan_line_comment`, which take `cpu` and use `cpu_select!`. The `avx2` module is now compiled for all x86_64 builds. `scan_block_comment_avx2` and `scan_line_comment_avx2` get `#[target_feature]` and take `_cpu: Avx2`. Their callers `lex_line_comment`, `lex_block_comment` and `jsx_skip_trivia` pass `cpu`.
Codegen: aarch64 is unchanged. With AVX2 flags, the AVX2 copies are unchanged, and `scan_block_comment_avx2` has 1 `mov` fewer. The generic copies of `carve`, `carve_jsx` and `jsx_skip_trivia` now call the generic scanners, as intended. In the x86_64 build, their AVX2 copies now call the AVX2 scanners. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Replace the cfg'd aliases in `find/mod.rs` with `#[inline(always)]` dispatchers `find` and `bracket_bits`, which take `cpu` and use `cpu_select!`. The `avx2` module is now compiled for all x86_64 builds. `find_avx2` and `bracket_bits_avx2` get `#[target_feature]` and take `cpu: Avx2`, which they pass to `any_of`. `any_of` takes `_cpu: Avx2`, and stays `#[inline(always)]` without `#[target_feature]`.
`carve_js`, `carve_jsx`, `find_line_terminator`, `scan_quoted`, `scan_regex`, `scan_tmpl_text` and `bracket_word` pass `cpu` to them.
`any_of` now builds its compares with a loop instead of `needles.map(..)`. `map` is `std` code, which is compiled without AVX2. In the x86_64 build, LLVM didn't inline `map` for 6 needles into the AVX2 copies of `carve_jsx` and `Brackets::word`, so each 32-byte block called `_mm256_set1_epi8` and `_mm256_cmpeq_epi8` out of line for each needle.
Codegen: aarch64 is unchanged. With AVX2 flags, the AVX2 copies have the same instructions as before, with registers and order shuffled in `carve`, `carve_jsx` and `delim_memo_opener`. Their generic copies, and the single-copy HTML comment functions, now use the generic versions, as intended. In the x86_64 build, the AVX2 copies now use the AVX2 versions, and `find_avx2` is inlined at every call site. The generic code's instruction count is unchanged.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `unicode_ws_len_at`, `line_terminator_after`, `line_break_in` and `block_comment_end`, keeping their inline attributes. `line_break_in` passes `cpu` to `line_terminator_after`. The others don't use `cpu` yet. Their callers pass `cpu`, and `Tokens::line_break_between`, `ident_is` and `skip_trivia_nl` now use theirs.
This changes inlining slightly. `line_terminator_after` is now inlined into `line_break_in`, which is called out of line instead. No other changes, apart from the new AVX2 versions in builds without AVX2 flags.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `kw_match_at`, keeping its `#[inline]` attribute. It doesn't use `cpu` yet. `Tokens::word_kw`, `gt_follower` and `type_list_legal` pass it `cpu`. Tests pass `BASELINE`.
No change to codegen, apart from the new AVX2 version in builds without AVX2 flags.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Resolved::closer`, `Closers::clear`, `Closers::get`, `Closers::set` and `Closers::set_args`, keeping their inline attributes. Their callers pass `cpu`. `DisambiguateState::restart` and `kill` now use theirs. None of these methods use `cpu` yet.
This changes inlining on x86_64. Each copy of `resolve_lists` and `resolve_groups` now inlines `Closers::set`, which was called out of line. They call the standard library's `HashMap::insert` out of line instead. These are cold paths, which run only for forward scans past their byte cap. On aarch64, `arrow_after_params` and `list_closer` change trivially.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to the free `prev_sig`, `Brackets::begin`, `Brackets::word`, `bracket_word`, `match_delim_back`, `delim_memo_opener`, `delim_slot` and `ident_is` in `common/walk.rs`, keeping their inline attributes. `Tokens::prev_sig`, `Tokens::ident_is`, `Tokens::match_delim_back`, `DisambiguateState::begin` and `carve_jsx` pass them `cpu`. `prev_sig`, `Brackets::begin`, `bracket_word`, `delim_slot` and `ident_is` don't use `cpu` yet.
This changes inlining on x86_64 only. `Brackets::begin` is inlined into each copy of `lex_raw_impl` again, which step C4 had undone. `Tokens::match_delim_back` is called out of line again, with `match_delim_back` inlined into it, as before step C7. No change on aarch64.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Tokens::base_kind`, `word_kw`, `ident_kw`, `next_start`, `next_sig`, `peek`, `prev_sig`, `prev_token`, `property_name`, `member_dot`, `numeric_dot`, `run_len`, `line_break_between`, `ident_is` and `match_delim_back`, keeping their inline attributes. Their callers in Levels 1 and 2 pass their `cpu`. `ends_decl`, `paren_close_is_regex` and `return_type_colon` now use theirs.
Level 3 is compiled only once, so the Level 3 functions which call these methods use `BASELINE`. So do `Tokens::template_head` and `Prev::is_member_dot`, which only Level 3 calls.
This changes inlining. `Tokens::ident_kw` and `Tokens::ident_is` are now called out of line from `prev_token`, `gt_run_split`, `type_args_at` and Level 3. In Level 3, `Walk::step` no longer inlines `Tokens::next_start`, `Walk::asi`, `Walk::end_region_for` or `Walk::continues_after_type_break`, so it shrinks from about 1,950 instructions to about 1,140. `not_operator_position` is unchanged on aarch64, and changes slightly on x86_64.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `lt_run_split`, `arrow_after_params`, `skip_trivia_fwd`, `list_closer`, `group_closer`, `past_raw`, `resolve_lists`, `kill`, `resolve_groups`, `skip_raw_literal`, `past_raw_string` and `raw_template_end` in `type_context/bytes.rs`, keeping their inline attributes. `skip_trivia_nl` takes `cpu`. It's `#[inline(always)]`, so it stays that way. `kill`, `past_raw_string` and `skip_trivia_nl` don't use `cpu` yet.
`coalesce`, `jsx_over_generic`, `gt_follower`, `type_list_legal` and `type_args_at` pass `cpu` to them. Level 3 is compiled only once, so `Walk::step_op` uses `BASELINE` for `lt_run_split`.
This changes codegen slightly. Register allocation changes in the cold `raw_template_end` on all platforms, and operand order in `skip_raw_literal` on aarch64.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `gt_follower`, `type_list_legal`, `keyword_type`, `type_args_at` and `list_is_type_args`, keeping their inline attributes. `list_in_any_context` passes `cpu` to `type_args_at`. `gt_follower` and `keyword_type` don't use `cpu` yet.
Level 3 is compiled only once, so its two callers, `Walk::less_than` and `Walk::type_word`, use `BASELINE`.
This changes inlining on x86_64. The AVX2 version of `type_args_at` now has only one caller, so it's inlined into the AVX2 version of `gt_run_split`, which grows from about 950 instructions to about 3,000. Its generic version is still called out of line, from Level 3 and from the generic version of `gt_run_split`. On aarch64, register allocation changes in `type_args_at`.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `DisambiguateState::begin` and `restart`. `lex_raw_impl` passes `cpu` to `begin`, and `coalesce` passes it to `restart`. Neither uses `cpu` yet.
This changes inlining on x86_64. `begin` was called out of line from both copies of `lex_raw_impl`. Now each copy inlines its own `begin`, including the small `Walks::restart`, and calls `Brackets::begin` out of line instead. This is setup code which runs once per lex. No change on aarch64.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `gt_run_split`, `run_shortcut`, `list_in_any_context`, `type_reference_head` and `return_type_colon`, keeping their inline attributes. `coalesce` and `glue_number` pass `cpu` to `gt_run_split`, which passes it down the chain. `return_type_colon` doesn't use `cpu` yet.
No change to codegen, apart from the new AVX2 versions in builds without AVX2 flags.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `jsx_over_generic`, keeping its `#[inline(never)]` attribute. `jsx_over_type_params` passes it `cpu`. It doesn't use `cpu` yet.
No change to codegen, apart from the new AVX2 version in builds without AVX2 flags.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `not_operator_position`, `ends_decl` and `paren_close_is_regex`. `lex_slash` and `carve_jsx` pass `cpu` to `not_operator_position`, which passes it on to the other two. `ends_decl` and `paren_close_is_regex` don't use `cpu` yet.
No change to codegen, apart from the new AVX2 versions in builds without AVX2 flags.
Part of effort to move to runtime CPU feature detection.
Mark `unicode_ws_len_at_unchecked` as `#[inline(always)]`, instead of compiling it separately for each CPU feature set. It's a thin wrapper around `unicode_ws_len`, which is `#[inline(always)]`, so it doesn't need a CPU type.
Why `#[inline(always)]`: Applying `cpu_versions!` to it, keeping its `#[inline]` attribute, made it no longer inlined on x86_64, into `misc_pre`, `jsx_skip_trivia` or `is_jsx_name_byte_at`. `misc_pre` calls it for every non-ASCII byte, so that put a function call in a hot loop. `#[inline(always)]` keeps it inlined everywhere, as it is now.
No change to codegen.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `scan_quoted`, `scan_regex`, `scan_tmpl_text`, `scan_ident_esc`, `scan_number` and `scan_fraction_exponent`, keeping their `#[inline]` attributes. Only `scan_number` uses `cpu` so far, to pass it to `scan_fraction_exponent`.
`lex_string`, `lex_template_segment`, `lex_regex` and `coalesce`'s `glue_number` pass `cpu` to them. `misc_pre_impl` takes `cpu` from `misc_pre`, to pass to `scan_ident_esc`. It's `#[inline(always)]`, so it stays that way.
This changes codegen. On x86_64, `scan_number` and `scan_fraction_exponent` are now inlined into `coalesce`, as they already were on aarch64. Before, each was called from both copies of `coalesce`. Register allocation and branch layout change slightly in `carve`, `carve_jsx` and `scan_regex`, on all platforms.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `parse_number`, `radix_digit` and `validate_number`, keeping their inline attributes. `parse_number` and `radix_digit` don't use `cpu` yet. `Lanes::push_number` passes `cpu` to `parse_number` and `validate_number`.
This changes inlining on x86_64, undoing step B8's change to `push_number`. `parse_number` is inlined into `push_number` again, and `push_number` is no longer inlined into `compress`, as before step B8. `push_number` is now the same as before step B1. On aarch64, branches in `push_number` change slightly.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Lanes::cook`, `cook_short_dispatch`, `cook_decode`, `push_cp` and `emit_oct`, keeping their inline attributes. `push_cp` and `emit_oct` don't use `cpu` yet. `push_string`, `push_template` and `push_atom` pass `cpu` to `cook`. Tests pass `BASELINE`.
Replace the 2 cfg'd versions of `cook_short` with an `#[inline(always)]` dispatcher which uses `cpu_select!`. `cook_short_avx2` gets `#[target_feature]` and takes `cpu: Avx2`, which it passes to `cook_decode`. It no longer needs `unsafe` blocks around the intrinsics which don't access memory. `cook_short_generic` gets `cpu_versions!`, because it passes `cpu` to `cook_decode`.
This changes inlining on x86_64, as step B7 did on aarch64. `cook` is now inlined into `push_string`, `push_template` and `push_atom`, which are no longer inlined into `carve` and `carve_jsx`. So the number of calls per string or template is unchanged. On aarch64, `cook_decode::<false, true>` is 52 instructions longer, with the same calls.
On x86_64, the AVX2 versions of `push_string`, `push_template` and `push_atom` now use `cook_short_avx2`. In builds with AVX2 flags, their generic versions use `cook_short_generic`, as intended for step D5.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Lanes::push_atom` and `push_number`, keeping their `#[inline]` attributes. They don't use `cpu` yet.
`compress`'s `emit_value` takes `cpu`. It's `#[inline(always)]`, so it stays that way. `lanes_post_avx2` passes it its `Avx2`. `lanes_post_generic` now needs a CPU type too, so it takes `cpu` and gets `cpu_versions!`. `lanes_post` passes it the CPU type from its `Neon` and `Generic` arms.
This changes inlining in `compress`. On x86_64, `push_number` is now inlined into `compress`, and `parse_number` is no longer inlined into `push_number`. `push_atom` is no longer inlined into `compress`. On aarch64, `push_atom` is no longer inlined into `compress`. Instead, `cook` is inlined into `push_atom`.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `Lanes::push_string`, `push_template`, `push_string_raw` and `push_regex_flags`, keeping their `#[inline]` attributes. They don't use `cpu` yet. `lex_string`, `lex_template_segment`, `lex_regex` and `carve_jsx` pass them `cpu`. A test passes `BASELINE`.
This changes codegen. On aarch64, `push_string` and `push_template` are no longer inlined into `carve` and `carve_jsx`. Instead, `cook` is inlined into them, so the number of calls per string or template is unchanged. On x86_64, register allocation changes in `carve` and `carve_jsx`.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `find_line_terminator`, keeping its `#[inline]` attribute. It doesn't use `cpu` yet. `carve_js`, `carve_jsx` and `jsx_skip_trivia` pass it `cpu`. The cold `lex_html_open_comment` and `lex_html_close_comment` are compiled only once, so they pass `BASELINE`.
No change to codegen.
Part of effort to move to runtime CPU feature detection.
`jsx_skip_trivia_fast` takes `cpu`. It's `#[inline(always)]`, so it stays that way. Apply `cpu_versions!` to the other functions in `carve/jsx/names.rs` which `carve_jsx` and `ts_angle_verdict` call, apart from the `#[inline(always)]` `is_jsx_name_byte`, which calls nothing that needs a CPU type. They keep their inline attributes. `is_jsx_name_byte_at`, `jsx_skip_trivia` and `jsx_name_end` don't use `cpu` yet.
This changes codegen slightly. On x86_64, `jsx_names_equal_fast` is inlined into `carve_jsx` again, as it was before step B1. On aarch64, `carve_jsx` makes 12 calls to `classify` instead of 16, and is about 200 instructions shorter. `jsx_name_next` changes trivially on all platforms.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `jsx_over_type_params` and `ts_angle_verdict`, keeping their `#[inline]` attributes. `carve_jsx` passes `cpu` to `jsx_over_type_params`, which passes it on to `ts_angle_verdict`. `ts_angle_verdict` doesn't use `cpu` yet.
This changes codegen on x86_64 only. `jsx_over_type_params` is inlined into `carve_jsx` again, as it was before step B1. No change on aarch64.
Part of effort to move to runtime CPU feature detection.
`lex_line_comment`, `lex_block_comment`, `html_open_comment_at` and `html_close_comment_at` take `cpu`. They're `#[inline(always)]`, so they stay that way. Apply `cpu_versions!` to the other functions in `carve/comments.rs` that they call, keeping their inline attributes. None of these functions use `cpu` for SIMD yet.
The cold `lex_html_open_comment` and `lex_html_close_comment` are compiled only once, so they pass `BASELINE`. `BASELINE` is now used outside tests, so remove the `expect(unused_imports)` on its re-exports.
This changes codegen slightly. `classify` now has 2 copies, so `str::trim_matches` (from `.trim()` in `is_property_key_annotation`) is no longer inlined into it. `html_close_at_line_start` differs slightly on all platforms. On aarch64, `carve` and `carve_jsx` lay out the code that pushes comments a little differently. No other functions are affected, apart from register allocation in the generic copy of `carve` on x86_64.
Part of effort to move to runtime CPU feature detection.
`lex_string`, `lex_template_segment`, `lex_slash` and `lex_regex` take `cpu`. They're `#[inline(always)]`, so they stay that way. Only `lex_slash` uses `cpu` so far, to pass it to `lex_regex`.
No change to codegen.
Part of effort to move to runtime CPU feature detection.
Apply `cpu_versions!` to `carve`, `carve_js` and `carve_jsx`, and pass `cpu` to `carve` from `lex_raw_impl`. `carve_js` and `carve_jsx` don't use `cpu` yet.
Part of effort to move to runtime CPU feature detection.
Mark `after`, `before` and `angles_before` in `disambiguate::context::scan` as `#[inline(never)]`. They are the entry points to disambiguate level 3, which will be compiled only once, not separately for each CPU feature set. So they must not be inlined into the separate copies of their callers.
This changes codegen on aarch64, x86_64, and x86_64 with AVX2. `before` and `angles_before` were previously inlined into `jsx_over_generic` and `gt_run_split`, along with `local_walk`. Now they're separate functions. `after` was already out of line. No other functions are affected.
Part of effort to move to runtime CPU feature detection.
Mark the cold functions `lex_html_open_comment`, `lex_html_close_comment` and `invalid_diags` as `#[inline(never)]`. They are single-copy code, which won't be compiled separately for each CPU feature set, so they must not be inlined into the separate copies of their callers.
No change to codegen. All 3 functions are already out of line on aarch64, x86_64, and x86_64 with AVX2.
Make the `cpu_select!` macro usable in const context, and `#[apply(cpu_versions)]` able to be put on const functions/methods.
Unclear at present if we'll need to specialize const functions for different CPUs, but it expands the capability of these macros at no cost.
Part of effort to move to runtime CPU feature detection.
Add a `BASELINE` const which represents the best feature set statically known to be supported on the current CPU. In practice this is `Neon` on aarch64, and `Generic` everywhere else.
Not used yet, but will be used in code which isn't compiled separately for different CPUs (disambiguate level 3) which needs to pass an `impl Cpu` to helper functions which are specialized by CPU features. These functions can use `BASELINE` as the `cpu` value.
Pure refactor - just moving code around.
Re-order functions in `find/avx2.rs`. `any_of` is a helper called by both `find_avx2` and `bracket_bits_avx2`, so move it to the bottom of the file, below both.
Pure refactor.
Follow-on after #28387. Move `unicode_ws_len` into `bytes` module, next to `unicode_ws_len_at` which does the same thing. Rename it to `unicode_ws_len_at_unchecked` to clarify the difference between the two.
`find` module now only contains the generic byte search infrastructure.
Pure refactor.
Follow-on after #28387. That PR replaced the many `find*` functions with a single generic `find`. Instead it introduced constants for the various byte search sets (`OPENER`, `OPENER8` etc).
Move these `const`s to live with the code that uses them (in `carve` and `scan`). `find` module only needs to contain the generic infrastructure.
Also rename them to make it clearer what they are.
overlookmotel