perf(parser): make pushing tokens faster (#19960)

Use `Vec::push_fast` method (introduced in #19959) for pushing to the tokens `Vec`. We allocate sufficient capacity upfront in the `Vec`, so it should never need to grow.

`Vec::push_fast` makes the "need to grow" path cold and `#[inline(never)]`, which should reduce the size of `Lexer::finish_next`, and help the branch predictor make accurate predictions.

+1% on benchmark for parsing with tokens in CodSpeed. *Possible* it'll have more impact in real world as CodSpeed doesn't take branch predictor into account in its simulations.
This commit is contained in:
overlookmotel committed 2026-03-03 13:55:44 +00:00
1 parent f83be3093e
commit 8ba61dd435
1 file changed
+5 -1
+5 -1
View File
@@ -302,7 +302,11 @@ impl<'a, C: Config> Lexer<'a, C> {
let token = self.token;
if self.config.tokens() {
match mode {
FinishTokenMode::Push => self.tokens.push(token),
FinishTokenMode::Push => {
// We allocated sufficient capacity in the `Vec` for all tokens at the start.
// `push_fast` is optimized for the "doesn't need to grow" case.
self.tokens.push_fast(token);
}
FinishTokenMode::Replace => {
debug_assert!(
self.tokens.last().is_some_and(|last| last.start() == token.start())