nprasad2077/oxc
1
0 Fork 0
mirror of https://github.com/oxc-project/oxc.git synced 2026-11-04 10:29:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(allocator): reduce instructions on allocation hot path (#21510)

Browse Source 
Perf optimization to `Arena`. Trim off instructions from allocation hot path (`try_alloc_layout_fast`).

Original implementation taken from `bumpalo` performed quite convoluted arithmetic to determine whether an allocation can be serviced within the current allocator chunk, or whether a new chunk needs to be added to the allocator.

Reduce these checks to 2 simple instructions.

The basis of the optimization is the invariants that:

1. A chunk's size can never be greater than `isize::MAX` bytes (because no allocation can be).
2. A`Layout`'s size can also never exceed `isize::MAX` bytes.

Utilizing these invariants, the bounds checks can be collapsed to a single subtraction and branch on the sign bit flag. More details in comments in the code.

When the allocation's requested alignment is larger than the arena's minimum alignment (which it is for all AST nodes, as they're all aligned on 8), this additionally collapses 2 branches into a single check.

On x86, the new version uses 1 less register. This is significant as this code gets inlined into call sites, potentially reducing stack spills.

| Path                  | Arch    | Old                                            | New                                            | Saved                                            |
| --------------------- | ------- | ---------------------------------------------- | ---------------------------------------------- | ------------------------------------------------ |
| Less                  | x86-64  | 7 instructions<br>1 branch<br>2 registers      | 5 instructions<br>1 branch<br>1 register       | -2 instructions<br>&nbsp;<br>-1 register         |
| Less                  | aarch64 | 8 instructions<br>1 branch<br>2 registers      | 7 instructions<br>1 branch<br>2 registers      | -1 instruction<br>&nbsp;<br>&nbsp;               |
| Less dynamic          | x86-64  | 10 instructions<br>1 branch<br>3 registers     | 6 instructions<br>1 branch<br>1 register       | -4 instructions<br>&nbsp;<br>-2 registers        |
| Less dynamic          | aarch64 | 10 instructions<br>1 branch<br>3 registers     | 8 instructions<br>1 branch<br>2 registers      | -2 instructions<br>&nbsp;<br>-1 register         |
| Equal                 | x86-64  | 7 instructions<br>1 branch<br>2 registers      | 5 instructions<br>1 branch<br>1 register       | -2 instructions<br>&nbsp;<br>-1 register         |
| Equal                 | aarch64 | 8 instructions<br>1 branch<br>2 registers      | 7 instructions<br>1 branch<br>2 registers      | -1 instruction<br>&nbsp;<br>&nbsp;               |
| Equal dynamic         | x86-64  | 8 instructions<br>1 branch<br>3 registers      | 5 instructions<br>1 branch<br>1 register       | -3 instructions<br>&nbsp;<br>-2 registers        |
| Equal dynamic         | aarch64 | 9 instructions<br>1 branch<br>3 registers      | 7 instructions<br>1 branch<br>2 registers      | -2 instructions<br>&nbsp;<br>-1 register         |
| Greater               | x86-64  | 9 instructions<br>2 branches<br>2 registers    | 6 instructions<br>1 branch<br>1 register       | -3 instructions<br>-1 branch<br>-1 register      |
| Greater               | aarch64 | 9 instructions<br>2 branches<br>2 registers    | 8 instructions<br>1 branch<br>2 registers      | -1 instruction<br>-1 branch<br>&nbsp;            |
| Greater dynamic       | x86-64  | 9 instructions<br>2 branches<br>2 registers    | 6 instructions<br>1 branch<br>1 register       | -3 instructions<br>-1 branch<br>-1 register      |
| Greater dynamic       | aarch64 | 9 instructions<br>1 branch<br>2 registers      | 8 instructions<br>1 branch<br>2 registers      | -1 instruction<br>&nbsp;<br>&nbsp;               |

1% perf improvement on all parser benchmarks.
This commit is contained in:
overlookmotel committed 2026-04-21 23:54:52 +00:00
1 parent 63eb98b6f2
commit 3cf7cef922
4 files changed
+916 -83
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

crates/oxc_allocator/src/arena/alloc_impl.rs
+239 -83
View File
@@ -6,19 +6,21 @@
use std::{
alloc::Layout,
cmp::Ordering,
cmp::max,
iter,
ptr::{self, NonNull},
};
use allocator_api2::alloc::{AllocError, Allocator};
use oxc_data_structures::assert_unchecked;
use super::{
Arena, CHUNK_FOOTER_SIZE, DEFAULT_CHUNK_SIZE_WITHOUT_FOOTER,
bumpalo_alloc::{Alloc as BumpaloAlloc, AllocErr},
utils::{
is_pointer_aligned_to, layout_from_size_align, oom, round_down_to, round_mut_ptr_down_to,
round_mut_ptr_up_to_unchecked, round_up_to, round_up_to_unchecked,
round_mut_ptr_up_to_unchecked, round_up_to,
},
};
@@ -60,91 +62,245 @@ impl<const MIN_ALIGN: usize> Arena<MIN_ALIGN> {
res
}
// Only `pub(super)` to expose it for unit tests
#[inline(always)]
fn try_alloc_layout_fast(&self, layout: Layout) -> Option<NonNull<u8>> {
// We don't need to check for ZSTs here since they will automatically be handled properly:
// the pointer will be bumped by zero bytes, modulo alignment.
// This keeps the fast path optimized for non-ZSTs, which are much more common.
unsafe {
let cursor_ptr = self.cursor_ptr.get().as_ptr();
let start_ptr = self.start_ptr.get().as_ptr();
pub(super) fn try_alloc_layout_fast(&self, layout: Layout) -> Option<NonNull<u8>> {
let cursor_ptr = self.cursor_ptr.get().as_ptr();
let start_ptr = self.start_ptr.get().as_ptr();
debug_assert!(
start_ptr <= cursor_ptr,
"start pointer {start_ptr:#p} should be less than or equal to bump pointer {cursor_ptr:#p}"
);
debug_assert!(
cursor_ptr <= self.current_chunk_footer.get().cast::<u8>().as_ptr(),
"bump pointer {cursor_ptr:#p} should be less than or equal to footer pointer {:#p}",
self.current_chunk_footer.get()
);
#[expect(clippy::checked_conversions)]
{
debug_assert!(
start_ptr <= cursor_ptr,
"start pointer {start_ptr:#p} should be less than or equal to bump pointer {cursor_ptr:#p}"
cursor_ptr.addr() - start_ptr.addr() <= isize::MAX as usize,
"distance between start pointer {start_ptr:#p} and bump pointer {cursor_ptr:#p} should be <= isize::MAX"
);
debug_assert!(
cursor_ptr <= self.current_chunk_footer.get().cast::<u8>().as_ptr(),
"bump pointer {cursor_ptr:#p} should be less than or equal to footer pointer {:#p}",
self.current_chunk_footer.get()
);
debug_assert!(
is_pointer_aligned_to(cursor_ptr, MIN_ALIGN),
"bump pointer {cursor_ptr:#p} should be aligned to the minimum alignment of {MIN_ALIGN:#x}"
);
// This `match` should be boiled away by LLVM: `MIN_ALIGN` is a constant and the layout's alignment
// is also constant in practice after inlining
let aligned_ptr = match layout.align().cmp(&MIN_ALIGN) {
Ordering::Less => {
// We need to round the size up to a multiple of `MIN_ALIGN` to preserve the minimum alignment.
// This might overflow since we cannot rely on `Layout`'s guarantees.
let aligned_size = round_up_to(layout.size(), MIN_ALIGN)?;
let capacity = (cursor_ptr as usize) - (start_ptr as usize);
if aligned_size > capacity {
return None;
}
cursor_ptr.wrapping_sub(aligned_size)
}
Ordering::Equal => {
// `Layout` guarantees that rounding the size up to its align cannot overflow
// (but does not guarantee that the size is initially a multiple of the alignment,
// which is why we need to do this rounding)
let aligned_size = round_up_to_unchecked(layout.size(), layout.align());
let capacity = (cursor_ptr as usize) - (start_ptr as usize);
if aligned_size > capacity {
return None;
}
cursor_ptr.wrapping_sub(aligned_size)
}
Ordering::Greater => {
// `Layout` guarantees that rounding the size up to its align cannot overflow
// (but does not guarantee that the size is initially a multiple of the alignment,
// which is why we need to do this rounding)
let aligned_size = round_up_to_unchecked(layout.size(), layout.align());
let aligned_ptr = round_mut_ptr_down_to(cursor_ptr, layout.align());
let capacity = (aligned_ptr as usize).wrapping_sub(start_ptr as usize);
if aligned_ptr < start_ptr || aligned_size > capacity {
return None;
}
aligned_ptr.wrapping_sub(aligned_size)
}
};
debug_assert!(
is_pointer_aligned_to(aligned_ptr, layout.align()),
"pointer {aligned_ptr:#p} should be aligned to layout alignment of {:#}",
layout.align()
);
debug_assert!(
is_pointer_aligned_to(aligned_ptr, MIN_ALIGN),
"pointer {aligned_ptr:#p} should be aligned to minimum alignment of {MIN_ALIGN:#}"
);
debug_assert!(
start_ptr <= aligned_ptr && aligned_ptr <= cursor_ptr,
"pointer {aligned_ptr:#p} should be in range {start_ptr:#p}..{cursor_ptr:#p}"
);
debug_assert!(!aligned_ptr.is_null());
let aligned_ptr = NonNull::new_unchecked(aligned_ptr);
self.cursor_ptr.set(aligned_ptr);
Some(aligned_ptr)
}
debug_assert!(
is_pointer_aligned_to(cursor_ptr, MIN_ALIGN),
"bump pointer {cursor_ptr:#p} should be aligned to the minimum alignment of {MIN_ALIGN:#x}"
);
// Compute `new_ptr` by:
//
// 1. Subtract `layout.size()` from `cursor_ptr`.
// 2. Round the result down to be aligned for both `MIN_ALIGN` and `layout.align()`.
// 3. Check if pointer is within bounds of current chunk.
//
// ------------
// # Invariants
// ------------
//
// This implementation relies on the following invariants:
//
// 1. `layout.size()` is always `<= isize::MAX`.
// 2. `layout.size()`, when rounded up to the nearest multiple of `layout.align()`, is always `<= isize::MAX`.
// 3. `cursor_ptr >= start_ptr`, because `cursor_ptr` is within the current chunk, which starts at `start_ptr`.
// 4. `cursor_ptr - start_ptr` is always `<= isize::MAX`, because they are both within the same allocation.
// 5. `MIN_ALIGN` is a power of two.
// 6. `cursor_ptr` is always aligned to `MIN_ALIGN`.
//
// Invariants 1 and 2 are invariants of `Layout` type.
// Invariant 4 is guaranteed by Rust.
// Invariants 3, 5 and 6 are guaranteed by `Arena` type.
//
// ---------------------
// # Computing `new_ptr`
// ---------------------
//
// ## `layout.align() > MIN_ALIGN`
//
// Subtract `layout.size()` from `cursor_ptr`, then round down to a multiple of `layout.align()` => `new_ptr`.
//
// `Layout`'s invariant is that `layout.size()`, when rounded up to the nearest multiple of `layout.align()`,
// is always `<= isize::MAX`.
// So the maximum subtraction (before rounding) is `isize::MAX - align + 1`.
//
// Rounding down to `align` can subtract at most a further `align - 1`.
// Therefore, maximum total subtraction including rounding is `isize::MAX - align + 1 + align - 1`
// = `isize::MAX`.
//
// ## `layout.align() <= MIN_ALIGN`
//
// Subtract `layout.size()` from `cursor_ptr`, then round down to a multiple of `MIN_ALIGN` => `new_ptr`.
//
// `Layout`'s invariant is that `layout.size() <= isize::MAX`.
// So the maximum subtraction (before rounding) is `isize::MAX`.
//
// Rounding down to `MIN_ALIGN` can subtract at most a further `MIN_ALIGN - 1`.
// However, since `cursor_ptr` is itself aligned to `MIN_ALIGN` (`Arena` invariant),
// the total subtraction (size + rounding padding) equals `round_up(layout.size(), MIN_ALIGN)`.
//
// `MIN_ALIGN` is a power of 2 and a `usize`, so at most `isize::MAX + 1`.
// `layout.size() <= isize::MAX` (invariant of `Layout`).
// So the maximum value of `round_up(layout.size(), MIN_ALIGN)` is `round_up(isize::MAX, isize::MAX + 1)`,
// which is `isize::MAX + 1`.
//
// Therefore maximum total subtraction including rounding is `isize::MAX + 1`.
//
// ## Combined
//
// Considering both cases, the maximum total subtraction is `isize::MAX + 1`.
//
// -------------------------
// # Detecting out of bounds
// -------------------------
//
// ## In bounds
//
// When `new_ptr` is within current chunk:
// * `start_ptr <= new_ptr <= cursor_ptr`.
// * Invariant: `cursor_ptr - start_ptr <= isize::MAX`.
// * So `new_ptr - start_ptr` does not wrap and is `<= isize::MAX`.
// * Therefore `new_ptr.wrapping_sub(start_ptr) <= isize::MAX`.
//
// ## Out of bounds
//
// When `new_ptr` is outside current chunk:
//
// (in all calculations that follow, "-" means wrapping subtraction)
//
// * `new_ptr = cursor_ptr - total_subtraction`.
// * `cursor_ptr = start_ptr + (cursor_ptr - start_ptr)` (trivially true).
// * So `new_ptr = start_ptr + (cursor_ptr - start_ptr) - total_subtraction`.
// * So `new_ptr - start_ptr = start_ptr + (cursor_ptr - start_ptr) - total_subtraction - start_ptr`
// => `new_ptr - start_ptr = (cursor_ptr - start_ptr) - total_subtraction`
// * Invariant: `cursor_ptr - start_ptr <= isize::MAX`.
// * `total_subtraction` is `<= isize::MAX + 1` (proved above).
// * `new_ptr.wrapping_sub(start_ptr)` always wraps (if `new_ptr >= start_ptr`, it would be in bounds).
// * In most extreme case:
// * `cursor_ptr == start_ptr` (chunk has no empty space).
// * `total_subtraction == isize::MAX + 1` (its maximum).
// * `new_ptr = cursor_ptr.wrapping_sub(isize::MAX + 1)`.
// * `new_ptr = start_ptr.wrapping_sub(isize::MAX + 1)`.
// * `new_ptr.wrapping_sub(start_ptr) = start_ptr.wrapping_sub(isize::MAX + 1).wrapping_sub(start_ptr)`
// = `0.wrapping_sub(isize::MAX + 1)`
// = `usize::MAX + 1 - (isize::MAX + 1)`
// = `isize::MAX + 1`
// * In all other out-of-bounds cases:
// * `cursor_ptr - start_ptr` is larger, or `total_subtraction` is smaller (or both).
// * Either change makes the wrapping subtraction wrap less (result is closer to `usize::MAX`).
// * So `new_ptr.wrapping_sub(start_ptr) > isize::MAX + 1`.
// * Therefore, in all out of bounds cases, `new_ptr.wrapping_sub(start_ptr) > isize::MAX`.
//
// ## Detection
//
// The difference between in bounds / out of bounds is:
// * In bounds: `new_ptr.wrapping_sub(start_ptr) <= isize::MAX`.
// * Out of bounds: `new_ptr.wrapping_sub(start_ptr) > isize::MAX`.
//
// Therefore `new_ptr.wrapping_sub(start_ptr) > isize::MAX` separates "in bounds" from "out of bounds".
//
// `> isize::MAX` is equivalent to checking if the top bit is set, which is very efficient in assembly.
// * `cmp new_ptr, start_ptr` subtracts and sets the sign flag (`SF` on x86, `N` on aarch64).
// * Branch on the sign flag (`js` on x86, `b.mi` on aarch64).
//
// Note: This calculation does *not* rely on pointers always being in bottom half of address space,
// which cannot be relied on, in particular on 32-bit platforms e.g. WASM.
//
// ---------------
// # Const folding
// ---------------
//
// `MIN_ALIGN` is a constant, and `layout.align()` is also constant in practice after inlining,
// so `max(layout.align(), MIN_ALIGN)` is folded down to a constant integer.
//
// For statically-known `layout.size()` (e.g. in `Arena::alloc`):
// We inform compiler that `cursor_ptr` is always aligned to `MIN_ALIGN` with an unchecked assertion.
// Combined with `round_mut_ptr_down_to`'s implementation as `p.wrapping_sub(p as usize & (divisor - 1))`,
// LLVM's known-bits analysis can:
//
// * compute the `& (divisor - 1)` term as a constant, and
// * fold `(cursor - constant_size) - constant_low_bits` into a single `cursor - constant`.
//
// Note: Writing the rounding as `p & !(divisor - 1)` instead of `p - (p & (divisor - 1))`
// would defeat this optimization — LLVM doesn't recognize the equivalent fold for the AND form.
//
// ---------------------------------------------------
// # Why not round up size first, then subtract after?
// ---------------------------------------------------
//
// For statically-known `layout.size()` (e.g. in `Arena::alloc`), due to const-folding,
// either version produces the same number of instructions.
//
// Subtracting first, then round down is preferable in 2 cases:
//
// 1. Dynamic size when `align < MIN_ALIGN`. e.g. `alloc_str` in an `Arena<MIN_ALIGN = 8>`:
// Rounding down is 1 instruction, whereas rounding up is 2 instructions.
// So the current version (subtract then round down) is 1 instruction shorter.
//
// 2. Over-aligned layouts e.g. size 8, align 16.
// Current version (subtract then round down) will only consume 8 bytes of arena memory if cursor
// is already positioned so that `cursor_ptr - 8` is aligned on 16.
// In comparison, rounding up size to multiple of 16 first, then subtracting, then rounding pointer down
// would consume 24 bytes.
//
// ------------------------
// # Zero-sized allocations
// ------------------------
//
// Zero-sized allocations require no special handling.
// The pointer will be bumped by zero bytes, modulo alignment.
// Avoiding a `layout.size() == 0` check keeps this code optimized for non-ZSTs, which are much more common.
//
// --------------
// # Instructions
// --------------
//
// This formulation overall produces the minimum possible instruction count for all use cases.
//
// Compared to `bumpalo`'s original implementation:
// * Fewer instructions on both x86-64 and aarch64.
// * When `layout.align() > MIN_ALIGN`, additionally removes a branch.
// * On x86-64, register usage reduced to only 1 register.
//
// x86: https://godbolt.org/z/r53d4W943
// aarch64: https://godbolt.org/z/KKeYnhGjx
// SAFETY: `cursor_ptr` is always aligned to `MIN_ALIGN` (invariant of `Arena`)
unsafe { assert_unchecked!(cursor_ptr.addr().is_multiple_of(MIN_ALIGN)) };
let new_ptr = cursor_ptr.wrapping_sub(layout.size());
let align = max(layout.align(), MIN_ALIGN);
let new_ptr = round_mut_ptr_down_to(new_ptr, align);
if new_ptr.addr().wrapping_sub(start_ptr.addr()) > (isize::MAX as usize) {
return None;
}
debug_assert!(
is_pointer_aligned_to(new_ptr, layout.align()),
"pointer {new_ptr:#p} should be aligned to layout alignment of {:#}",
layout.align()
);
debug_assert!(
is_pointer_aligned_to(new_ptr, MIN_ALIGN),
"pointer {new_ptr:#p} should be aligned to minimum alignment of {MIN_ALIGN:#}"
);
debug_assert!(
start_ptr <= new_ptr && new_ptr <= cursor_ptr,
"pointer {new_ptr:#p} should be in range {start_ptr:#p}..{cursor_ptr:#p}"
);
debug_assert!(!new_ptr.is_null());
// SAFETY: `start_ptr` is non-null, so if `new_ptr` was null, `new_ptr.addr().wrapping_sub(start_ptr.addr())`
// above would have wrapped around, and we already exited
let new_ptr = unsafe { NonNull::new_unchecked(new_ptr) };
// Update cursor
self.cursor_ptr.set(new_ptr);
// Return the pointer
Some(new_ptr)
}
/// Slow path allocation for when we need to allocate a new chunk, because there isn't enough room
crates/oxc_allocator/src/arena/mod.rs
+2
View File
@@ -31,6 +31,8 @@ mod from_raw_parts;
#[cfg(test)]
mod tests;
#[cfg(test)]
mod tests_alloc;
/// An arena to allocate into.
///
crates/oxc_allocator/src/arena/tests_alloc.rs
+666
View File
@@ -0,0 +1,666 @@
//! Unit tests for `try_alloc_layout_fast` bounds check.
//!
//! Uses `TestArena` to construct arenas with controlled `cursor_ptr` and `start_ptr` values,
//! including addresses in the top half of address space (top bit set) which can't be tested
//! via integration tests on any 64-bit platform we run CI on.
//!
//! All addresses are derived from `isize::MAX` so the same tests work on both 32-bit and 64-bit.
use std::{alloc::Layout, cell::Cell, cmp, mem::ManuallyDrop, ptr::NonNull};
use super::{Arena, CHUNK_ALIGN, ChunkFooter};
/// First address in the top half of address space (top bit set).
/// `0x8000_0000` on 32-bit, `0x8000_0000_0000_0000` on 64-bit.
const TOP: usize = (isize::MAX as usize) + 1;
/// Test wrapper that creates an `Arena` with fake `cursor_ptr` and `start_ptr` values.
/// Wrapped in `ManuallyDrop` so the `Arena` is never dropped (which would try to dealloc the fake pointers).
struct TestArena<const MIN_ALIGN: usize = 1> {
inner: ManuallyDrop<Arena<MIN_ALIGN>>,
}
impl<const MIN_ALIGN: usize> TestArena<MIN_ALIGN> {
/// Create a `TestArena` with `cursor_ptr` and `start_ptr` set to the given addresses.
/// `current_chunk_footer` is pointed at `cursor` (satisfying `cursor <= footer` debug assert).
/// The footer pointer is never dereferenced in `try_alloc_layout_fast`.
fn new(start: usize, cursor: usize) -> Self {
// Check input is valid
assert!(
start.is_multiple_of(CHUNK_ALIGN),
"start {start:#x} must be a multiple of CHUNK_ALIGN ({CHUNK_ALIGN})"
);
assert!(cursor >= start, "cursor {cursor:#x} must be >= start {start:#x}");
let capacity = cursor - start;
let chunk_align = cmp::max(MIN_ALIGN, CHUNK_ALIGN);
assert!(
Layout::from_size_align(capacity, chunk_align).is_ok(),
"capacity {capacity} (cursor {cursor:#x} - start {start:#x}) with align {chunk_align} must be a valid `Layout`",
);
// Construct arena
let cursor_ptr = NonNull::new(cursor as *mut u8).unwrap();
let start_ptr = NonNull::new(start as *mut u8).unwrap();
let arena = Arena {
cursor_ptr: Cell::new(cursor_ptr),
current_chunk_footer: Cell::new(cursor_ptr.cast::<ChunkFooter>()),
start_ptr: Cell::new(start_ptr),
can_grow: false,
#[cfg(all(feature = "track_allocations", not(feature = "disable_track_allocations")))]
stats: crate::tracking::AllocationStats::default(),
};
Self { inner: ManuallyDrop::new(arena) }
}
fn try_alloc_layout_fast(&self, layout: Layout) -> Option<NonNull<u8>> {
self.inner.try_alloc_layout_fast(layout)
}
}
// --- Bottom half of address space ---
#[test]
fn bottom_half_enough_room() {
let arena = TestArena::<1>::new(0x1000, 0x2000);
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x2000 - 32);
}
#[test]
fn bottom_half_exact_fit() {
let arena = TestArena::<1>::new(0x1000, 0x1020); // 32 bytes capacity
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1000);
}
#[test]
fn bottom_half_not_enough_room() {
let arena = TestArena::<1>::new(0x1000, 0x1010); // 16 bytes capacity
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
#[test]
fn bottom_half_zero_capacity() {
let arena = TestArena::<1>::new(0x1000, 0x1000);
let layout = Layout::from_size_align(1, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Top half of address space (top bit set) ---
#[test]
fn top_half_enough_room() {
let start = TOP + 0x1000;
let cursor = start + 0x1000;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 32);
}
#[test]
fn top_half_exact_fit() {
let start = TOP + 0x1000;
let cursor = start + 32;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, start);
}
#[test]
fn top_half_not_enough_room() {
let start = TOP + 0x1000;
let cursor = start + 16;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
#[test]
fn top_half_zero_capacity() {
let start = TOP + 0x1000;
let arena = TestArena::<1>::new(start, start);
let layout = Layout::from_size_align(1, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Wrapping (allocation size larger than cursor address) ---
#[test]
fn wrapping_sub_wraps_bottom_half() {
let arena = TestArena::<1>::new(0x100, 0x200);
let layout = Layout::from_size_align(0x1000, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
#[test]
fn wrapping_sub_wraps_across_midpoint() {
// Cursor is just past the top-bit boundary; subtracting pushes into the bottom half
let start = TOP;
let cursor = TOP + 0x10;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(0x20, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- ZST ---
#[test]
fn zst_zero_capacity() {
let arena = TestArena::<1>::new(0x1000, 0x1000);
let layout = Layout::from_size_align(0, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1000);
}
#[test]
fn zst_top_half() {
let addr = TOP + 0x1000;
let arena = TestArena::<1>::new(addr, addr);
let layout = Layout::from_size_align(0, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, addr);
}
// --- Greater path: align > MIN_ALIGN ---
#[test]
fn greater_enough_room() {
// Cursor at 0x1037, rounds down to 0x1030. 0x30 bytes from start. Fits 16 bytes.
let arena = TestArena::<1>::new(0x1000, 0x1037);
let layout = Layout::from_size_align(16, 8).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1030 - 16);
assert_eq!(result.unwrap().as_ptr() as usize % 8, 0);
}
#[test]
fn greater_not_enough_room() {
// Cursor at 0x1017, rounds down to 0x1010. Only 0x10 bytes from start. 32 doesn't fit.
let arena = TestArena::<1>::new(0x1000, 0x1017);
let layout = Layout::from_size_align(32, 8).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
#[test]
fn greater_rounds_below_start() {
// Start at 0x1010, cursor at 0x1013. Subtracting size 4 gives 0x100F, then rounding down
// to align 16 gives 0x1000 < start. Must reject.
let arena = TestArena::<1>::new(0x1010, 0x1013);
let layout = Layout::from_size_align(4, 16).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
#[test]
fn greater_top_half() {
// Use an address in top half that's aligned to 8
let start = TOP + 0x1000;
let cursor = start + 0x100;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(32, 8).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 32);
assert_eq!(result.unwrap().as_ptr() as usize % 8, 0);
}
#[test]
fn greater_top_half_rounds_below_start() {
// Start at TOP+0x10, cursor at TOP+0x13. Subtracting size 4 gives TOP+0xF, then rounding
// down to align 16 gives TOP+0 < start. Must reject.
let start = TOP + 0x10;
let cursor = TOP + 0x13;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(4, 16).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Maximum allocation sizes ---
/// Equal path: aligned_size = isize::MAX (the maximum valid Layout size with align 1).
/// Always rejected — no chunk can hold isize::MAX bytes.
#[test]
fn max_size_equal_path() {
let layout = Layout::from_size_align(isize::MAX as usize, 1).unwrap();
// Bottom half
let arena = TestArena::<1>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
// Top half
let arena = TestArena::<1>::new(TOP + 0x1000, TOP + 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Less path: layout.size() = isize::MAX with align 1 in Arena<8>.
/// aligned_size = round_up(isize::MAX, 8) = isize::MAX + 1. This is the maximum possible
/// aligned_size. Must be rejected.
#[test]
fn max_size_less_path_rounds_above_isize_max() {
let layout = Layout::from_size_align(isize::MAX as usize, 1).unwrap();
// Bottom half
let arena = TestArena::<8>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
// Top half
let arena = TestArena::<8>::new(TOP + 0x1000, TOP + 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Less path: layout.size() = isize::MAX with align 1 in Arena<16>.
/// aligned_size = round_up(isize::MAX, 16) = isize::MAX + 1. Must be rejected.
#[test]
fn max_size_less_path_min_align_16() {
let layout = Layout::from_size_align(isize::MAX as usize, 1).unwrap();
let arena = TestArena::<16>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
let arena = TestArena::<16>::new(TOP + 0x1000, TOP + 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Greater path: max Layout size for align 8 is isize::MAX - 7 (so rounded size fits in isize::MAX).
/// Must be rejected since no chunk is that big.
#[test]
fn max_size_greater_path() {
// isize::MAX - 7 is the largest size where Layout::from_size_align(size, 8) succeeds
let layout = Layout::from_size_align((isize::MAX as usize) - 7, 8).unwrap();
let arena = TestArena::<1>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
let arena = TestArena::<1>::new(TOP + 0x1000, TOP + 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Less path with size just below the rounding boundary.
/// size = isize::MAX - 7 with MIN_ALIGN=8 rounds to isize::MAX - 7 (already a multiple of 8).
/// Still too large for any real chunk.
#[test]
fn near_max_size_less_path_already_aligned() {
let layout = Layout::from_size_align((isize::MAX as usize) - 7, 1).unwrap();
let arena = TestArena::<8>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Boundary case: Greater path with the largest valid Layout for align=256.
/// Layout::from_size_align(isize::MAX - 255, 256) gives the maximum size whose round-up to 256
/// fits in isize::MAX. Combined with worst-case padding (cursor positioned at `start mod 256 = 240`,
/// the maximum cursor mod 256 achievable when start is CHUNK_ALIGN(16)-aligned), total subtraction
/// reaches isize::MAX - 15. The bounds check must reject this.
#[test]
fn boundary_greater_path_max_layout_with_max_padding() {
let layout = Layout::from_size_align((isize::MAX as usize) - 255, 256).unwrap();
// Empty chunk with start chosen so cursor mod 256 = 240 (max for CHUNK_ALIGN-aligned start)
let arena = TestArena::<1>::new(0xF0, 0xF0);
assert!(arena.try_alloc_layout_fast(layout).is_none());
// Same in top half
let arena = TestArena::<1>::new(TOP + 0xF0, TOP + 0xF0);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Boundary case: total subtraction (size + padding) equals exactly `isize::MAX + 1`.
/// This is the worst case for the bounds check — uses the `>` (not `>=`) discrimination.
/// With MIN_ALIGN=16 and size=isize::MAX-14, padding=15 → total = 2^63 = isize::MAX + 1.
#[test]
fn boundary_total_subtraction_equals_isize_max_plus_one() {
let layout = Layout::from_size_align((isize::MAX as usize) - 14, 1).unwrap();
// Empty chunk — cursor == start. Worst case for the bounds check.
let arena = TestArena::<16>::new(0x1000, 0x1000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
// Same in top half.
let arena = TestArena::<16>::new(TOP + 0x1000, TOP + 0x1000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Less path with size that rounds to exactly isize::MAX.
/// size = isize::MAX - 8 with MIN_ALIGN=8: round_up(isize::MAX - 8, 8) = isize::MAX - 7.
/// (isize::MAX = 0x7FFF...FFFF, isize::MAX - 8 = 0x7FFF...FFF7, rounded = 0x7FFF...FFF8 = isize::MAX - 7)
#[test]
fn near_max_size_less_path_rounds_to_near_max() {
let layout = Layout::from_size_align((isize::MAX as usize) - 8, 1).unwrap();
let arena = TestArena::<8>::new(0x1000, 0x2000);
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Multiple consecutive allocations ---
/// Verify cursor moves correctly across multiple allocations.
#[test]
fn consecutive_allocations() {
let arena = TestArena::<1>::new(0x1000, 0x1100); // 256 bytes
let layout = Layout::from_size_align(32, 1).unwrap();
let p1 = arena.try_alloc_layout_fast(layout).unwrap();
assert_eq!(p1.as_ptr() as usize, 0x1100 - 32);
let p2 = arena.try_alloc_layout_fast(layout).unwrap();
assert_eq!(p2.as_ptr() as usize, 0x1100 - 64);
let p3 = arena.try_alloc_layout_fast(layout).unwrap();
assert_eq!(p3.as_ptr() as usize, 0x1100 - 96);
}
/// Multiple allocations in top half.
#[test]
fn consecutive_allocations_top_half() {
let start = TOP + 0x1000;
let cursor = start + 0x100;
let arena = TestArena::<8>::new(start, cursor);
let layout = Layout::from_size_align(32, 8).unwrap();
let p1 = arena.try_alloc_layout_fast(layout).unwrap();
assert_eq!(p1.as_ptr() as usize, cursor - 32);
let p2 = arena.try_alloc_layout_fast(layout).unwrap();
assert_eq!(p2.as_ptr() as usize, cursor - 64);
}
/// Fill chunk to exact capacity, then fail.
#[test]
fn fill_then_fail() {
let arena = TestArena::<1>::new(0x1000, 0x1040); // 64 bytes
let layout = Layout::from_size_align(16, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_some()); // 48 left
assert!(arena.try_alloc_layout_fast(layout).is_some()); // 32 left
assert!(arena.try_alloc_layout_fast(layout).is_some()); // 16 left
assert!(arena.try_alloc_layout_fast(layout).is_some()); // 0 left
assert!(arena.try_alloc_layout_fast(layout).is_none()); // fail
}
// --- Off-by-one at capacity boundary ---
/// Capacity = size - 1: one byte short, must fail.
#[test]
fn off_by_one_too_small() {
let arena = TestArena::<1>::new(0x1000, 0x101F); // 31 bytes
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
/// Capacity = size: exact fit, must succeed.
#[test]
fn off_by_one_exact() {
let arena = TestArena::<1>::new(0x1000, 0x1020); // 32 bytes
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_some());
}
/// Capacity = size + 1: one byte spare, must succeed.
#[test]
fn off_by_one_just_enough() {
let arena = TestArena::<1>::new(0x1000, 0x1021); // 33 bytes
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_some());
}
/// Off-by-one in top half.
#[test]
fn off_by_one_top_half() {
let start = TOP + 0x1000;
let arena_small = TestArena::<1>::new(start, start + 31);
let arena_exact = TestArena::<1>::new(start, start + 32);
let layout = Layout::from_size_align(32, 1).unwrap();
assert!(arena_small.try_alloc_layout_fast(layout).is_none());
assert!(arena_exact.try_alloc_layout_fast(layout).is_some());
}
// --- Chunk spanning the midpoint (start in bottom half, cursor in top half) ---
/// Start near the bottom of the top-bit boundary, cursor just past it.
/// This is a pathological case that can't happen on real 64-bit platforms
/// (user space never spans the midpoint), but is valid on 32-bit.
#[test]
fn chunk_spans_midpoint() {
let start = TOP - 0x100;
let cursor = TOP + 0x100;
let arena = TestArena::<1>::new(start, cursor);
// Small allocation should succeed — cursor is in top half, result stays in range
let layout = Layout::from_size_align(16, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 16);
}
/// Same as above but allocation is too big.
#[test]
fn chunk_spans_midpoint_not_enough_room() {
let start = TOP - 0x10;
let cursor = TOP + 0x10; // only 32 bytes capacity
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(64, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Addresses near usize::MAX ---
/// Chunk near the very top of address space.
#[test]
fn near_usize_max() {
// Place chunk near the end of address space.
// start must be CHUNK_ALIGN-aligned and capacity must be a valid Layout.
let cursor = usize::MAX - (CHUNK_ALIGN - 1); // align down to CHUNK_ALIGN
let start = cursor - 0x100;
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(16, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 16);
}
/// Near usize::MAX, allocation too big.
#[test]
fn near_usize_max_not_enough_room() {
let cursor = usize::MAX - (CHUNK_ALIGN - 1);
let start = cursor - 0x20; // 32 bytes
let arena = TestArena::<1>::new(start, cursor);
let layout = Layout::from_size_align(64, 1).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Minimum allocation (size 1) ---
#[test]
fn size_one_bottom_half() {
let arena = TestArena::<1>::new(0x1000, 0x1001);
let layout = Layout::from_size_align(1, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1000);
}
#[test]
fn size_one_top_half() {
let start = TOP + 0x1000;
let arena = TestArena::<1>::new(start, start + 1);
let layout = Layout::from_size_align(1, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, start);
}
// --- ZST with large alignment (Greater path) ---
/// ZST with align > MIN_ALIGN. Cursor must round down to alignment but size is 0,
/// so new_ptr == aligned_cursor. Should succeed as long as aligned_cursor >= start.
#[test]
fn zst_greater_path() {
// cursor at 0x1037, rounds down to 0x1030. Size 0, so new_ptr = 0x1030.
// 0x1030 >= 0x1000 (start), so succeeds.
let arena = TestArena::<1>::new(0x1000, 0x1037);
let layout = Layout::from_size_align(0, 8).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1030);
}
/// ZST with align > MIN_ALIGN where rounding cursor down lands exactly on start.
#[test]
fn zst_greater_path_exact_start() {
// cursor at 0x1003, rounds down to 0x1000 == start. Size 0, new_ptr = start. Succeeds.
let arena = TestArena::<1>::new(0x1000, 0x1003);
let layout = Layout::from_size_align(0, 16).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1000);
}
// --- Greater path: cursor already aligned ---
/// When cursor is already aligned to layout.align(), no padding is wasted.
#[test]
fn greater_cursor_already_aligned() {
// cursor at 0x1040 (already aligned to 8). No rounding needed.
let arena = TestArena::<1>::new(0x1000, 0x1040);
let layout = Layout::from_size_align(16, 8).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1040 - 16);
}
// --- Arena<16> (maximum MIN_ALIGN) ---
#[test]
fn arena16_equal_path() {
let arena = TestArena::<16>::new(0x1000, 0x1100);
let layout = Layout::from_size_align(48, 16).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1100 - 48);
assert_eq!(result.unwrap().as_ptr() as usize % 16, 0);
}
#[test]
fn arena16_less_path() {
let arena = TestArena::<16>::new(0x1000, 0x1100);
// 5 bytes with align 1, rounds up to 16
let layout = Layout::from_size_align(5, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1100 - 16);
}
#[test]
fn arena16_top_half() {
let start = TOP + 0x1000;
let cursor = start + 0x100;
let arena = TestArena::<16>::new(start, cursor);
let layout = Layout::from_size_align(32, 16).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 32);
assert_eq!(result.unwrap().as_ptr() as usize % 16, 0);
}
// --- Lowest valid start address ---
#[test]
fn lowest_valid_start() {
// Start at CHUNK_ALIGN (the lowest non-zero aligned address)
let cursor = CHUNK_ALIGN + 0x100;
let arena = TestArena::<1>::new(CHUNK_ALIGN, cursor);
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 32);
}
#[test]
fn lowest_valid_start_not_enough_room() {
// Start at CHUNK_ALIGN (the lowest non-zero aligned address)
let arena = TestArena::<1>::new(CHUNK_ALIGN, CHUNK_ALIGN + 16);
let layout = Layout::from_size_align(32, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_none());
}
// --- Less path: align < MIN_ALIGN ---
#[test]
fn less_rounds_size_up() {
let arena = TestArena::<8>::new(0x1000, 0x1100);
// 3 bytes with align 1, but MIN_ALIGN=8 rounds size up to 8
let layout = Layout::from_size_align(3, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, 0x1100 - 8);
}
#[test]
fn less_top_half() {
let start = TOP + 0x1000;
let cursor = start + 0x100;
let arena = TestArena::<8>::new(start, cursor);
let layout = Layout::from_size_align(3, 1).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 8);
}
// --- Equal path: align == MIN_ALIGN ---
#[test]
fn equal_top_half() {
let start = TOP + 0x1000;
let cursor = start + 0x100;
let arena = TestArena::<8>::new(start, cursor);
let layout = Layout::from_size_align(32, 8).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 32);
}
#[test]
fn equal_not_enough_room_top_half() {
let start = TOP + 0x1000;
let cursor = start + 16; // only 16 bytes, need 32
let arena = TestArena::<8>::new(start, cursor);
let layout = Layout::from_size_align(32, 8).unwrap();
assert!(arena.try_alloc_layout_fast(layout).is_none());
}
// --- Over-aligned layout ---
#[test]
fn over_aligned() {
let start = 0x1000;
let cursor = start + 0x1008;
let arena = TestArena::<8>::new(start, cursor);
let layout = Layout::from_size_align(8, 16).unwrap();
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 8);
assert_eq!(result.unwrap().as_ptr() as usize % 16, 0);
let result = arena.try_alloc_layout_fast(layout);
assert!(result.is_some());
assert_eq!(result.unwrap().as_ptr() as usize, cursor - 24);
assert_eq!(result.unwrap().as_ptr() as usize % 16, 0);
}
crates/oxc_allocator/src/arena/utils.rs
+9
View File
@@ -43,6 +43,15 @@ pub fn round_down_to(n: usize, divisor: usize) -> usize {
}
/// Same as `round_down_to` but preserves pointer provenance.
///
/// # Implementation note
///
/// The exact implementation here is important.
///
/// `Arena::try_alloc_layout_fast` relies on it using `wrapping_sub`.
/// `ptr.map_addr(|addr| addr & !(divisor - 1))` would be semantically equivalent,
/// but would prevent LLVM from folding 2 subtractions into 1 in `try_alloc_layout_fast`.
/// See comment in that method for more details.
#[inline]
pub fn round_mut_ptr_down_to(ptr: *mut u8, divisor: usize) -> *mut u8 {
debug_assert!(divisor > 0);