Joao Martins 31b912de13 mm/gup: decrement head page once for group of subpages ... Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Link: https://lkml.kernel.org/r/20210212130843.13865-3-joao.m.martins@oracle.com Signed-off-by: Joao Martins <joao.m.martins@oracle.com> Reviewed-by: John Hubbard <jhubbard@nvidia.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Christoph Hellwig <hch@infradead.org> Cc: Doug Ledford <dledford@redhat.com> Cc: Matthew Wilcox <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2021-04-30 11:20:37 -07:00
..
kasan	arm64 updates for 5.13:	2021-04-26 10:25:03 -07:00
kfence	kfence: make compatible with kmemleak	2021-03-25 09:22:55 -07:00
backing-dev.c	mm/backing-dev.c: use might_alloc()	2021-02-26 09:41:01 -08:00
balloon_compaction.c
cleancache.c
cma_debug.c
cma.c	mm: cma: print region name on failure	2021-02-26 09:41:00 -08:00
cma.h	mm: cma: use CMA_MAX_NAME to define the length of cma name array	2020-09-01 09:19:43 +02:00
compaction.c	mm, compaction: make fast_isolate_freepages() stay within zone	2021-02-24 13:38:34 -08:00
debug_page_ref.c
debug_vm_pgtable.c	mm/debug_vm_pgtable/basic: iterate over entire protection_map[]	2021-02-24 13:38:27 -08:00
debug.c	mm/debug: improve memcg debugging	2021-02-24 13:38:27 -08:00
dmapool.c	mm/dmapool: use might_alloc()	2021-02-26 09:41:01 -08:00
early_ioremap.c	mm/early_ioremap.c: use __func__ instead of function name	2021-02-26 09:41:02 -08:00
fadvise.c	mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED	2020-10-13 18:38:29 -07:00
failslab.c
filemap.c	mm/filemap: update stale comment	2021-04-30 11:20:37 -07:00
frontswap.c	mm/frontswap: mark various intentional data races	2020-08-14 19:56:56 -07:00
gup_test.c	mm/gup_test.c: mark gup_test_init as __init function	2020-12-15 12:13:38 -08:00
gup_test.h	selftests/vm: gup_test: introduce the dump_pages() sub-test	2020-12-15 12:13:38 -08:00
gup.c	mm/gup: decrement head page once for group of subpages	2021-04-30 11:20:37 -07:00
highmem.c	mm/highmem: fix CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP	2021-03-25 09:22:55 -07:00
hmm.c
huge_memory.c	mm/memcg: rename mem_cgroup_split_huge_fixup to split_page_memcg and add nr_pages argument	2021-03-13 11:27:31 -08:00
hugetlb_cgroup.c	hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings	2021-03-25 09:22:55 -07:00
hugetlb.c	hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings	2021-03-25 09:22:55 -07:00
hwpoison-inject.c	mm,hwpoison-inject: don't pin for hwpoison_filter	2020-10-16 11:11:16 -07:00
init-mm.c	mm/gup: prevent gup_fast from racing with COW during fork	2020-12-15 12:13:39 -08:00
internal.h	Network filesystem helper library	2021-04-27 13:08:12 -07:00
interval_tree.c
ioremap.c
Kconfig	block: remove BLK_BOUNCE_ISA support	2021-04-06 09:28:17 -06:00
Kconfig.debug	mm, page_poison: remove CONFIG_PAGE_POISONING_ZERO	2020-12-15 12:13:46 -08:00
khugepaged.c	mm,thp,shmem: make khugepaged obey tmpfs mount flags	2021-02-26 09:40:59 -08:00
kmemleak.c	mm/kmemleak.c: fix a typo	2021-04-30 11:20:36 -07:00
ksm.c	mm: cleanup kstrto*() usage	2020-12-15 12:13:47 -08:00
list_lru.c	mm/list_lru.c: remove kvfree_rcu_local()	2021-02-24 13:38:30 -08:00
maccess.c
madvise.c	mm/madvise: replace ptrace attach requirement for process_madvise	2021-03-13 11:27:30 -08:00
Makefile	mm: add Kernel Electric-Fence infrastructure	2021-02-26 09:41:02 -08:00
mapping_dirty_helpers.c	mm/mapping_dirty_helpers: guard hugepage pud's usage	2021-04-16 16:10:37 -07:00
memblock.c	memblock: remove return value of memblock_free_all()	2021-02-22 13:01:23 -08:00
memcontrol.c	mm: page-writeback: simplify memcg handling in test_clear_page_writeback()	2021-04-30 11:20:37 -07:00
memfd.c
memory_hotplug.c	arm64: mte: Map hotplugged memory as Normal Tagged	2021-03-10 10:56:46 +00:00
memory-failure.c	mm: fix memory_failure() handling of dax-namespace metadata	2021-02-26 09:41:00 -08:00
memory.c	mm: fix race by making init_zero_pfn() early_initcall	2021-03-30 09:46:12 -07:00
mempolicy.c	mm/mempolicy: use helper range_in_vma() in queue_pages_test_walk()	2021-02-24 13:38:34 -08:00
mempool.c	kasan: move _RET_IP_ to inline wrappers	2021-02-24 13:38:31 -08:00
memremap.c	mm/memory_hotplug: prevalidate the address range being added with platform	2021-02-26 09:41:00 -08:00
memtest.c
migrate.c	mm: memcg: add swapcache stat for memcg v2	2021-02-24 13:38:29 -08:00
mincore.c	inode: make init and permission helpers idmapped mount aware	2021-01-24 14:27:16 +01:00
mlock.c	mm/mlock: stop counting mlocked pages when none vma is found	2021-02-26 09:41:01 -08:00
mm_init.c	mm: fix fall-through warnings for Clang	2020-12-15 12:13:47 -08:00
mmap_lock.c	mm: mmap_lock: add tracepoints around lock acquisition	2020-12-15 12:13:41 -08:00
mmap.c	arm64: Support execute-only permissions with Enhanced PAN	2021-03-26 09:37:23 +00:00
mmu_gather.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
mmu_notifier.c	mm/mmu_notifiers: ensure range_end() is paired with range_start()	2021-03-25 09:22:55 -07:00
mmzone.c	mm/lru: replace pgdat lru_lock with lruvec lock	2020-12-15 14:48:04 -08:00
mprotect.c	mm/mprotect.c: optimize error detection in do_mprotect_pkey()	2021-02-24 13:38:30 -08:00
mremap.c	mm: mremap: unlink anon_vmas when mremap with MREMAP_DONTUNMAP success	2021-02-24 13:38:30 -08:00
msync.c	mm/msync: exit early when the flags is an MS_ASYNC and start < vm_start	2021-04-30 11:20:37 -07:00
nommu.c	mm/nommu: Fix return type of filemap_map_pages()	2021-01-28 14:10:31 +00:00
oom_kill.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
page_alloc.c	init_on_alloc: Optimize static branches	2021-04-08 14:05:19 +02:00
page_counter.c	mm/page_counter: use page_counter_read in page_counter_set_max	2020-12-15 12:13:40 -08:00
page_ext.c	mm: fix some spelling mistakes in comments	2020-12-15 22:46:19 -08:00
page_idle.c	mm: page_idle_get_page() does not need lru_lock	2020-12-15 14:48:03 -08:00
page_io.c	swap: fix swapfile read/write offset	2021-03-02 17:25:46 -07:00
page_isolation.c	mm/page_isolation: do not isolate the max order page	2020-12-15 12:13:45 -08:00
page_owner.c	mm: page_owner: detect page_owner recursion via task_struct	2021-04-30 11:20:36 -07:00
page_poison.c	mm: page_poison: print page info when corruption is caught	2021-04-30 11:20:36 -07:00
page_reporting.c	mm/page_reporting: use list_entry_is_head() in page_reporting_cycle()	2021-02-24 13:38:30 -08:00
page_reporting.h
page_vma_mapped.c	mm/page_vma_mapped.c: add colon to fix kernel-doc markups error for check_pte	2020-12-15 12:13:41 -08:00
page-writeback.c	mm: page-writeback: simplify memcg handling in test_clear_page_writeback()	2021-04-30 11:20:37 -07:00
pagewalk.c
percpu-internal.h	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
percpu-km.c
percpu-stats.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
percpu-vm.c
percpu.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
pgalloc-track.h
pgtable-generic.c	mm/pgtable-generic.c: optimize the VM_BUG_ON condition in pmdp_huge_clear_flush()	2021-02-24 13:38:30 -08:00
process_vm_access.c	mm/process_vm_access.c: include compat.h	2021-01-12 18:12:54 -08:00
ptdump.c	mm: ptdump: fix build failure	2021-04-16 16:10:37 -07:00
readahead.c	mm: Implement readahead_control pageset expansion	2021-04-23 10:14:29 +01:00
rmap.c	mm/rmap: correct obsolete comment of page_get_anon_vma()	2021-02-26 09:41:01 -08:00
rodata_test.c	mm/rodata_test.c: fix missing function declaration	2020-08-21 09:52:53 -07:00
shmem.c	shmem: allow reporting fanotify events with file handles on tmpfs	2021-04-19 16:03:48 +02:00
shuffle.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
shuffle.h
slab_common.c	mm/slab_common: provide "slab_merge" option for !IS_ENABLED(CONFIG_SLAB_MERGE_DEFAULT) builds	2021-04-30 11:20:36 -07:00
slab.c	Merge branch 'for-mingo-rcu' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu	2021-04-11 14:31:43 +02:00
slab.h	RCU changes for this cycle were:	2021-04-28 12:00:13 -07:00
slob.c	mm: Don't build mm_dump_obj() on CONFIG_PRINTK=n kernels	2021-03-08 14:18:46 -08:00
slub.c	mm/slub.c: trivial typo fixes	2021-04-30 11:20:36 -07:00
sparse-vmemmap.c
sparse.c	mm/memory_hotplug: guard more declarations by CONFIG_MEMORY_HOTPLUG	2020-10-16 11:11:18 -07:00
swap_cgroup.c
swap_slots.c	mm/swap_slots.c: remove redundant NULL check	2021-02-24 13:38:28 -08:00
swap_state.c	mm: add FGP_ENTRY	2021-02-26 09:40:59 -08:00
swap.c	mm: remove pagevec_lookup_entries	2021-02-26 09:40:59 -08:00
swapfile.c	swap: fix swapfile read/write offset	2021-03-02 17:25:46 -07:00
truncate.c	mm: remove pagevec_lookup_entries	2021-02-26 09:40:59 -08:00
usercopy.c
userfaultfd.c
util.c	mm: move page_mapping_file to pagemap.h	2021-04-30 11:20:37 -07:00
vmacache.c
vmalloc.c	mm: Don't build mm_dump_obj() on CONFIG_PRINTK=n kernels	2021-03-08 14:18:46 -08:00
vmpressure.c
vmscan.c	mm/vmscan: restore zone_reclaim_mode ABI	2021-02-24 13:38:34 -08:00
vmstat.c	mm/vmstat.c: erase latency in vmstat_shepherd	2021-02-26 09:41:00 -08:00
workingset.c	mm: workingset: clarify eviction order and distance calculation	2021-02-24 13:38:34 -08:00
z3fold.c	z3fold: prevent reclaim/free race for headless pages	2021-03-25 09:22:55 -07:00
zbud.c	mm: set the sleep_mapped to true for zbud and z3fold	2021-02-26 09:41:01 -08:00
zpool.c	mm/zswap: add the flag can_sleep_mapped	2021-02-26 09:41:01 -08:00
zsmalloc.c	mm/zsmalloc.c: use page_private() to access page->private	2021-02-26 09:41:01 -08:00
zswap.c	mm/zswap: add the flag can_sleep_mapped	2021-02-26 09:41:01 -08:00