mirror of
https://github.com/torvalds/linux.git
synced 2024-12-24 11:51:27 +00:00
ad59baa316
Slab allocators have been guaranteeing natural alignment for
power-of-two sizes since commit 59bb47985c ("mm, sl[aou]b: guarantee
natural alignment for kmalloc(power-of-two)"), while any other sizes are
guaranteed to be aligned only to ARCH_KMALLOC_MINALIGN bytes (although
in practice are aligned more than that in non-debug scenarios).
Rust's allocator API specifies size and alignment per allocation, which
have to satisfy the following rules, per Alice Ryhl [1]:
1. The alignment is a power of two.
2. The size is non-zero.
3. When you round up the size to the next multiple of the alignment,
then it must not overflow the signed type isize / ssize_t.
In order to map this to kmalloc()'s guarantees, some requested
allocation sizes have to be padded to the next power-of-two size [2].
For example, an allocation of size 96 and alignment of 32 will be padded
to an allocation of size 128, because the existing kmalloc-96 bucket
doesn't guarantee alignent above ARCH_KMALLOC_MINALIGN. Without slab
debugging active, the layout of the kmalloc-96 slabs however naturally
align the objects to 32 bytes, so extending the size to 128 bytes is
wasteful.
To improve the situation we can extend the kmalloc() alignment
guarantees in a way that
1) doesn't change the current slab layout (and thus does not increase
internal fragmentation) when slab debugging is not active
2) reduces waste in the Rust allocator use case
3) is a superset of the current guarantee for power-of-two sizes.
The extended guarantee is that alignment is at least the largest
power-of-two divisor of the requested size. For power-of-two sizes the
largest divisor is the size itself, but let's keep this case documented
separately for clarity.
For current kmalloc size buckets, it means kmalloc-96 will guarantee
alignment of 32 bytes and kmalloc-196 will guarantee 64 bytes.
This covers the rules 1 and 2 above of Rust's API as long as the size is
a multiple of the alignment. The Rust layer should now only need to
round up the size to the next multiple if it isn't, while enforcing the
rule 3.
Implementation-wise, this changes the alignment calculation in
create_boot_cache(). While at it also do the calulation only for caches
with the SLAB_KMALLOC flag, because the function is also used to create
the initial kmem_cache and kmem_cache_node caches, where no alignment
guarantee is necessary.
In the Rust allocator's krealloc_aligned(), remove the code that padded
sizes to the next power of two (suggested by Alice Ryhl) as it's no
longer necessary with the new guarantees.
Reported-by: Alice Ryhl <aliceryhl@google.com>
Reported-by: Boqun Feng <boqun.feng@gmail.com>
Link: https://lore.kernel.org/all/CAH5fLggjrbdUuT-H-5vbQfMazjRDpp2%2Bk3%3DYhPyS17ezEqxwcw@mail.gmail.com/ [1]
Link: https://lore.kernel.org/all/CAH5fLghsZRemYUwVvhk77o6y1foqnCeDzW4WZv6ScEWna2+_jw@mail.gmail.com/ [2]
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Acked-by: Roman Gushchin <roman.gushchin@linux.dev>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
75 lines
3.0 KiB
Rust
75 lines
3.0 KiB
Rust
// SPDX-License-Identifier: GPL-2.0
|
|
|
|
//! Allocator support.
|
|
|
|
use super::{flags::*, Flags};
|
|
use core::alloc::{GlobalAlloc, Layout};
|
|
use core::ptr;
|
|
|
|
struct KernelAllocator;
|
|
|
|
/// Calls `krealloc` with a proper size to alloc a new object aligned to `new_layout`'s alignment.
|
|
///
|
|
/// # Safety
|
|
///
|
|
/// - `ptr` can be either null or a pointer which has been allocated by this allocator.
|
|
/// - `new_layout` must have a non-zero size.
|
|
pub(crate) unsafe fn krealloc_aligned(ptr: *mut u8, new_layout: Layout, flags: Flags) -> *mut u8 {
|
|
// Customized layouts from `Layout::from_size_align()` can have size < align, so pad first.
|
|
let layout = new_layout.pad_to_align();
|
|
|
|
// Note that `layout.size()` (after padding) is guaranteed to be a multiple of `layout.align()`
|
|
// which together with the slab guarantees means the `krealloc` will return a properly aligned
|
|
// object (see comments in `kmalloc()` for more information).
|
|
let size = layout.size();
|
|
|
|
// SAFETY:
|
|
// - `ptr` is either null or a pointer returned from a previous `k{re}alloc()` by the
|
|
// function safety requirement.
|
|
// - `size` is greater than 0 since it's from `layout.size()` (which cannot be zero according
|
|
// to the function safety requirement)
|
|
unsafe { bindings::krealloc(ptr as *const core::ffi::c_void, size, flags.0) as *mut u8 }
|
|
}
|
|
|
|
unsafe impl GlobalAlloc for KernelAllocator {
|
|
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
|
// SAFETY: `ptr::null_mut()` is null and `layout` has a non-zero size by the function safety
|
|
// requirement.
|
|
unsafe { krealloc_aligned(ptr::null_mut(), layout, GFP_KERNEL) }
|
|
}
|
|
|
|
unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) {
|
|
unsafe {
|
|
bindings::kfree(ptr as *const core::ffi::c_void);
|
|
}
|
|
}
|
|
|
|
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
|
|
// SAFETY:
|
|
// - `new_size`, when rounded up to the nearest multiple of `layout.align()`, will not
|
|
// overflow `isize` by the function safety requirement.
|
|
// - `layout.align()` is a proper alignment (i.e. not zero and must be a power of two).
|
|
let layout = unsafe { Layout::from_size_align_unchecked(new_size, layout.align()) };
|
|
|
|
// SAFETY:
|
|
// - `ptr` is either null or a pointer allocated by this allocator by the function safety
|
|
// requirement.
|
|
// - the size of `layout` is not zero because `new_size` is not zero by the function safety
|
|
// requirement.
|
|
unsafe { krealloc_aligned(ptr, layout, GFP_KERNEL) }
|
|
}
|
|
|
|
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
|
// SAFETY: `ptr::null_mut()` is null and `layout` has a non-zero size by the function safety
|
|
// requirement.
|
|
unsafe { krealloc_aligned(ptr::null_mut(), layout, GFP_KERNEL | __GFP_ZERO) }
|
|
}
|
|
}
|
|
|
|
#[global_allocator]
|
|
static ALLOCATOR: KernelAllocator = KernelAllocator;
|
|
|
|
// See <https://github.com/rust-lang/rust/pull/86844>.
|
|
#[no_mangle]
|
|
static __rust_no_alloc_shim_is_unstable: u8 = 0;
|