Joonsoo Kim 18726ca8b3 mm/slab: fix the theoretical race by holding proper lock ... While processing concurrent allocation, SLAB could be contended a lot because it did a lots of work with holding a lock. This patchset try to reduce the number of critical section to reduce lock contention. Major changes are lockless decision to allocate more slab and lockless cpu cache refill from the newly allocated slab. Below is the result of concurrent allocation/free in slab allocation benchmark made by Christoph a long time ago. I make the output simpler. The number shows cycle count during alloc/free respectively so less is better. * Before Kmalloc N*alloc N*free(32): Average=365/806 Kmalloc N*alloc N*free(64): Average=452/690 Kmalloc N*alloc N*free(128): Average=736/886 Kmalloc N*alloc N*free(256): Average=1167/985 Kmalloc N*alloc N*free(512): Average=2088/1125 Kmalloc N*alloc N*free(1024): Average=4115/1184 Kmalloc N*alloc N*free(2048): Average=8451/1748 Kmalloc N*alloc N*free(4096): Average=16024/2048 * After Kmalloc N*alloc N*free(32): Average=344/792 Kmalloc N*alloc N*free(64): Average=347/882 Kmalloc N*alloc N*free(128): Average=390/959 Kmalloc N*alloc N*free(256): Average=393/1067 Kmalloc N*alloc N*free(512): Average=683/1229 Kmalloc N*alloc N*free(1024): Average=1295/1325 Kmalloc N*alloc N*free(2048): Average=2513/1664 Kmalloc N*alloc N*free(4096): Average=4742/2172 It shows that performance improves greatly (roughly more than 50%) for the object class whose size is more than 128 bytes. This patch (of 11): If we don't hold neither the slab_mutex nor the node lock, node's shared array cache could be freed and re-populated. If __kmem_cache_shrink() is called at the same time, it will call drain_array() with n->shared without holding node lock so problem can happen. This patch fix the situation by holding the node lock before trying to drain the shared array. In addition, add a debug check to confirm that n->shared access race doesn't exist. Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: Jesper Dangaard Brouer <brouer@redhat.com> Cc: Christoph Lameter <cl@linux.com> Cc: Pekka Enberg <penberg@kernel.org> Cc: David Rientjes <rientjes@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2016-05-19 19:12:14 -07:00
..
kasan	mm, kasan: fix compilation for CONFIG_SLAB	2016-04-01 17:03:37 -05:00
backing-dev.c	writeback: fix the wrong congested state variable definition	2016-03-31 12:26:25 -06:00
balloon_compaction.c	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial	2016-03-17 21:38:27 -07:00
bootmem.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
cleancache.c	cleancache: constify cleancache_ops structure	2016-01-27 09:09:57 -05:00
cma_debug.c	mm/cma_debug: correct size input to bitmap function	2015-07-17 16:39:54 -07:00
cma.c	mm/cma.c: suppress warning	2015-11-05 19:34:48 -08:00
cma.h	mm: cma: mark cma_bitmap_maxno() inline in header	2015-08-14 15:56:32 -07:00
compaction.c	mm: fix kcompactd hang during memory offlining	2016-05-05 17:38:53 -07:00
debug_page_ref.c	mm/page_ref: add tracepoint to track down page reference manipulation	2016-03-17 15:09:34 -07:00
debug.c	mm: introduce page reference manipulation functions	2016-03-17 15:09:34 -07:00
dmapool.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
early_ioremap.c	mm/early_ioremap: use offset_in_page macro	2015-11-05 19:34:48 -08:00
fadvise.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
failslab.c	mm: fault-inject take over bootstrap kmem_cache check	2016-03-15 16:55:16 -07:00
filemap.c	fs: simplify the generic_write_sync prototype	2016-05-01 19:58:39 -04:00
frame_vector.c	mm/gup: Switch all callers of get_user_pages() to not pass tsk/mm	2016-02-16 10:11:12 +01:00
frontswap.c	frontswap: allow multiple backends	2015-06-24 17:49:45 -07:00
gup.c	Merge branch 'mm-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-04-14 19:31:34 -07:00
highmem.c
huge_memory.c	mm: thp: calculate the mapcount correctly for THP pages during WP faults	2016-05-12 15:52:50 -07:00
hugetlb_cgroup.c	mm: make compound_head() robust	2015-11-06 17:50:42 -08:00
hugetlb.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
hwpoison-inject.c	hwpoison: use page_cgroup_ino for filtering by memcg	2015-09-10 13:29:01 -07:00
init-mm.c
internal.h	mm, oom: introduce oom reaper	2016-03-25 16:37:42 -07:00
interval_tree.c
Kconfig	Merge branch 'mm-pkeys-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-03-20 19:08:56 -07:00
Kconfig.debug	mm/page_ref: add tracepoint to track down page reference manipulation	2016-03-17 15:09:34 -07:00
kmemcheck.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak-test.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak.c	mm: coalesce split strings	2016-03-17 15:09:34 -07:00
ksm.c	ksm: fix conflict between mmput and scan_get_next_rmap_item	2016-05-12 15:52:50 -07:00
list_lru.c	mm: memcontrol: move kmem accounting code to CONFIG_MEMCG	2016-01-20 17:09:18 -08:00
maccess.c	mm/maccess.c: actually return -EFAULT from strncpy_from_unsafe	2015-11-05 19:34:48 -08:00
madvise.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
Makefile	mm, kasan: SLAB support	2016-03-25 16:37:42 -07:00
memblock.c	mm: coalesce split strings	2016-03-17 15:09:34 -07:00
memcontrol.c	memcg: relocate charge moving from ->attach to ->post_attach	2016-04-25 15:45:14 -04:00
memory_hotplug.c	mm: coalesce split strings	2016-03-17 15:09:34 -07:00
memory-failure.c	mm/memory-failure: fix race with compound page split/merge	2016-04-28 19:34:04 -07:00
memory.c	mm: thp: calculate the mapcount correctly for THP pages during WP faults	2016-05-12 15:52:50 -07:00
mempolicy.c	Merge branch 'mm-pkeys-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-03-20 19:08:56 -07:00
mempool.c	mm, kasan: add GFP flags to KASAN API	2016-03-25 16:37:42 -07:00
memtest.c	memtest: remove unused header files	2015-09-08 15:35:28 -07:00
migrate.c	mm/hwpoison: fix wrong num_poisoned_pages accounting	2016-04-28 19:34:04 -07:00
mincore.c	mm, fs: remove remaining PAGE_CACHE_* and page_cache_{get,release} usage	2016-04-04 10:41:08 -07:00
mlock.c	mm: fix mlock accouting	2016-01-21 17:20:51 -08:00
mm_init.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
mmap.c	Merge branch 'mm-pkeys-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-03-20 19:08:56 -07:00
mmu_context.c	mm/mmu_context, sched/core: Fix mmu_context.h assumption	2016-04-28 11:44:19 +02:00
mmu_notifier.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
mmzone.c	mm/mmzone.c: memmap_valid_within() can be boolean	2016-01-14 16:00:49 -08:00
mprotect.c	mm/mprotect.c: don't imply PROT_EXEC on non-exec fs	2016-03-22 15:36:02 -07:00
mremap.c	mm: cleanup *pte_alloc* interfaces	2016-03-17 15:09:34 -07:00
msync.c	mm/msync: use offset_in_page macro	2015-11-05 19:34:48 -08:00
nobootmem.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
nommu.c	Merge branch 'mm-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-04-14 19:31:34 -07:00
oom_kill.c	oom, oom_reaper: do not enqueue task if it is on the oom_reaper_list head	2016-04-01 17:03:37 -05:00
page_alloc.c	mm: update min_free_kbytes from khugepaged after core initialization	2016-05-05 17:38:53 -07:00
page_counter.c	mm: page_counter: let page_counter_try_charge() return bool	2015-11-05 19:34:48 -08:00
page_ext.c	mm/page_poisoning.c: allow for zero poisoning	2016-03-15 16:55:16 -07:00
page_idle.c	mm: add page_check_address_transhuge() helper	2016-01-15 17:56:32 -08:00
page_io.c	Merge branch 'work.preadv2' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2016-05-17 15:05:23 -07:00
page_isolation.c	mm/page_isolation.c: fix the function comments	2016-04-01 17:03:37 -05:00
page_owner.c	mm: coalesce split strings	2016-03-17 15:09:34 -07:00
page_poison.c	mm/page_poisoning.c: allow for zero poisoning	2016-03-15 16:55:16 -07:00
page-writeback.c	Merge branch 'for-linus' of git://git.kernel.dk/linux-block	2016-05-06 13:08:35 -07:00
pagewalk.c	thp: rename split_huge_page_pmd() to split_huge_pmd()	2016-01-15 17:56:32 -08:00
percpu-km.c	mm: percpu: use pr_fmt to prefix output	2016-03-17 15:09:34 -07:00
percpu-vm.c
percpu.c	mm: percpu: use pr_fmt to prefix output	2016-03-17 15:09:34 -07:00
pgtable-generic.c	mm/thp/migration: switch from flush_tlb_range to flush_pmd_tlb_range	2016-03-17 15:09:34 -07:00
process_vm_access.c	mm/gup: Introduce get_user_pages_remote()	2016-02-16 10:04:09 +01:00
quicklist.c	fix Christoph's email addresses	2016-03-17 15:09:34 -07:00
readahead.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
rmap.c	Merge branch 'PAGE_CACHE_SIZE-removal'	2016-04-04 10:50:24 -07:00
shmem.c	parallel lookups machinery, part 2	2016-05-02 19:49:26 -04:00
slab_common.c	mm, kasan: add GFP flags to KASAN API	2016-03-25 16:37:42 -07:00
slab.c	mm/slab: fix the theoretical race by holding proper lock	2016-05-19 19:12:14 -07:00
slab.h	mm, kasan: add GFP flags to KASAN API	2016-03-25 16:37:42 -07:00
slob.c	mm: slab: free kmem_cache_node after destroy sysfs file	2016-02-18 16:23:24 -08:00
slub.c	mm, kasan: add GFP flags to KASAN API	2016-03-25 16:37:42 -07:00
sparse-vmemmap.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
sparse.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
swap_cgroup.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
swap_state.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
swap.c	thp: keep huge zero page pinned until tlb flush	2016-04-28 19:34:04 -07:00
swapfile.c	mm: thp: calculate the mapcount correctly for THP pages during WP faults	2016-05-12 15:52:50 -07:00
truncate.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
userfaultfd.c	mm, fs: get rid of PAGE_CACHE_* and page_cache_{get,release} macros	2016-04-04 10:41:08 -07:00
util.c	Merge branch 'mm-pkeys-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2016-03-20 19:08:56 -07:00
vmacache.c	mm/vmacache: inline vmacache_valid_mm()	2015-11-05 19:34:48 -08:00
vmalloc.c	mm/vmalloc: use PAGE_ALIGNED() to check PAGE_SIZE alignment	2016-03-17 15:09:34 -07:00
vmpressure.c	mm/vmpressure.c: fix subtree pressure detection	2016-02-03 08:28:43 -08:00
vmscan.c	mm: wake kcompactd before kswapd's short sleep	2016-04-28 19:34:04 -07:00
vmstat.c	thp, vmstats: count deferred split events	2016-03-17 15:09:34 -07:00
workingset.c	mm: workingset: make shadow node shrinker memcg aware	2016-03-17 15:09:34 -07:00
zbud.c	mm/zbud.c: use list_last_entry() instead of list_tail_entry()	2016-01-15 11:40:52 -08:00
zpool.c	mm: zsmalloc: constify struct zs_pool name	2015-11-06 17:50:42 -08:00
zsmalloc.c	zsmalloc: fix zs_can_compact() integer overflow	2016-05-09 17:40:59 -07:00
zswap.c	mm/zswap: provide unique zpool name	2016-05-05 17:38:53 -07:00