Joao Martins 8745d7f634 mm/gup: add compound page list iterator ... Patch series "mm/gup: page unpining improvements", v4. This series improves page unpinning, with an eye on improving MR deregistration for big swaths of memory (which is bound by the page unpining), particularly: 1) Decrement the head page by @ntails and thus reducing a lot the number of atomic operations per compound page. This is done by comparing individual tail pages heads, and counting number of consecutive tails on which they match heads and based on that update head page refcount. Should have a visible improvement in all page (un)pinners which use compound pages 2) Introducing a new API for unpinning page ranges (to avoid the trick in the previous item and be based on math), and use that in RDMA ib_mem_release (used for mr deregistration). Performance improvements: unpin_user_pages() for hugetlbfs and THP improves ~3x (through gup_test) and RDMA MR dereg improves ~4.5x with the new API. See patches 2 and 4 for those. This patch (of 4): Add a helper that iterates over head pages in a list of pages. It essentially counts the tails until the next page to process has a different head that the current. This is going to be used by unpin_user_pages() family of functions, to batch the head page refcount updates once for all passed consecutive tail pages. Link: https://lkml.kernel.org/r/20210212130843.13865-1-joao.m.martins@oracle.com Link: https://lkml.kernel.org/r/20210212130843.13865-2-joao.m.martins@oracle.com Signed-off-by: Joao Martins <joao.m.martins@oracle.com> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> Reviewed-by: John Hubbard <jhubbard@nvidia.com> Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Cc: Doug Ledford <dledford@redhat.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Christoph Hellwig <hch@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2021-04-30 11:20:37 -07:00
..
kasan	arm64 updates for 5.13:	2021-04-26 10:25:03 -07:00
kfence	kfence: make compatible with kmemleak	2021-03-25 09:22:55 -07:00
backing-dev.c	mm/backing-dev.c: use might_alloc()	2021-02-26 09:41:01 -08:00
balloon_compaction.c
cleancache.c
cma_debug.c
cma.c	mm: cma: print region name on failure	2021-02-26 09:41:00 -08:00
cma.h	mm: cma: use CMA_MAX_NAME to define the length of cma name array	2020-09-01 09:19:43 +02:00
compaction.c	mm, compaction: make fast_isolate_freepages() stay within zone	2021-02-24 13:38:34 -08:00
debug_page_ref.c
debug_vm_pgtable.c	mm/debug_vm_pgtable/basic: iterate over entire protection_map[]	2021-02-24 13:38:27 -08:00
debug.c	mm/debug: improve memcg debugging	2021-02-24 13:38:27 -08:00
dmapool.c	mm/dmapool: use might_alloc()	2021-02-26 09:41:01 -08:00
early_ioremap.c	mm/early_ioremap.c: use __func__ instead of function name	2021-02-26 09:41:02 -08:00
fadvise.c	mm, fadvise: improve the expensive remote LRU cache draining after FADV_DONTNEED	2020-10-13 18:38:29 -07:00
failslab.c
filemap.c	mm/filemap: update stale comment	2021-04-30 11:20:37 -07:00
frontswap.c	mm/frontswap: mark various intentional data races	2020-08-14 19:56:56 -07:00
gup_test.c	mm/gup_test.c: mark gup_test_init as __init function	2020-12-15 12:13:38 -08:00
gup_test.h	selftests/vm: gup_test: introduce the dump_pages() sub-test	2020-12-15 12:13:38 -08:00
gup.c	mm/gup: add compound page list iterator	2021-04-30 11:20:37 -07:00
highmem.c	mm/highmem: fix CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP	2021-03-25 09:22:55 -07:00
hmm.c
huge_memory.c	mm/memcg: rename mem_cgroup_split_huge_fixup to split_page_memcg and add nr_pages argument	2021-03-13 11:27:31 -08:00
hugetlb_cgroup.c	hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings	2021-03-25 09:22:55 -07:00
hugetlb.c	hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings	2021-03-25 09:22:55 -07:00
hwpoison-inject.c	mm,hwpoison-inject: don't pin for hwpoison_filter	2020-10-16 11:11:16 -07:00
init-mm.c	mm/gup: prevent gup_fast from racing with COW during fork	2020-12-15 12:13:39 -08:00
internal.h	Network filesystem helper library	2021-04-27 13:08:12 -07:00
interval_tree.c
ioremap.c
Kconfig	block: remove BLK_BOUNCE_ISA support	2021-04-06 09:28:17 -06:00
Kconfig.debug	mm, page_poison: remove CONFIG_PAGE_POISONING_ZERO	2020-12-15 12:13:46 -08:00
khugepaged.c	mm,thp,shmem: make khugepaged obey tmpfs mount flags	2021-02-26 09:40:59 -08:00
kmemleak.c	mm/kmemleak.c: fix a typo	2021-04-30 11:20:36 -07:00
ksm.c	mm: cleanup kstrto*() usage	2020-12-15 12:13:47 -08:00
list_lru.c	mm/list_lru.c: remove kvfree_rcu_local()	2021-02-24 13:38:30 -08:00
maccess.c
madvise.c	mm/madvise: replace ptrace attach requirement for process_madvise	2021-03-13 11:27:30 -08:00
Makefile	mm: add Kernel Electric-Fence infrastructure	2021-02-26 09:41:02 -08:00
mapping_dirty_helpers.c	mm/mapping_dirty_helpers: guard hugepage pud's usage	2021-04-16 16:10:37 -07:00
memblock.c	memblock: remove return value of memblock_free_all()	2021-02-22 13:01:23 -08:00
memcontrol.c	mm: page-writeback: simplify memcg handling in test_clear_page_writeback()	2021-04-30 11:20:37 -07:00
memfd.c
memory_hotplug.c	arm64: mte: Map hotplugged memory as Normal Tagged	2021-03-10 10:56:46 +00:00
memory-failure.c	mm: fix memory_failure() handling of dax-namespace metadata	2021-02-26 09:41:00 -08:00
memory.c	mm: fix race by making init_zero_pfn() early_initcall	2021-03-30 09:46:12 -07:00
mempolicy.c	mm/mempolicy: use helper range_in_vma() in queue_pages_test_walk()	2021-02-24 13:38:34 -08:00
mempool.c	kasan: move _RET_IP_ to inline wrappers	2021-02-24 13:38:31 -08:00
memremap.c	mm/memory_hotplug: prevalidate the address range being added with platform	2021-02-26 09:41:00 -08:00
memtest.c
migrate.c	mm: memcg: add swapcache stat for memcg v2	2021-02-24 13:38:29 -08:00
mincore.c	inode: make init and permission helpers idmapped mount aware	2021-01-24 14:27:16 +01:00
mlock.c	mm/mlock: stop counting mlocked pages when none vma is found	2021-02-26 09:41:01 -08:00
mm_init.c	mm: fix fall-through warnings for Clang	2020-12-15 12:13:47 -08:00
mmap_lock.c	mm: mmap_lock: add tracepoints around lock acquisition	2020-12-15 12:13:41 -08:00
mmap.c	arm64: Support execute-only permissions with Enhanced PAN	2021-03-26 09:37:23 +00:00
mmu_gather.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
mmu_notifier.c	mm/mmu_notifiers: ensure range_end() is paired with range_start()	2021-03-25 09:22:55 -07:00
mmzone.c	mm/lru: replace pgdat lru_lock with lruvec lock	2020-12-15 14:48:04 -08:00
mprotect.c	mm/mprotect.c: optimize error detection in do_mprotect_pkey()	2021-02-24 13:38:30 -08:00
mremap.c	mm: mremap: unlink anon_vmas when mremap with MREMAP_DONTUNMAP success	2021-02-24 13:38:30 -08:00
msync.c	mm/msync: exit early when the flags is an MS_ASYNC and start < vm_start	2021-04-30 11:20:37 -07:00
nommu.c	mm/nommu: Fix return type of filemap_map_pages()	2021-01-28 14:10:31 +00:00
oom_kill.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
page_alloc.c	init_on_alloc: Optimize static branches	2021-04-08 14:05:19 +02:00
page_counter.c	mm/page_counter: use page_counter_read in page_counter_set_max	2020-12-15 12:13:40 -08:00
page_ext.c	mm: fix some spelling mistakes in comments	2020-12-15 22:46:19 -08:00
page_idle.c	mm: page_idle_get_page() does not need lru_lock	2020-12-15 14:48:03 -08:00
page_io.c	swap: fix swapfile read/write offset	2021-03-02 17:25:46 -07:00
page_isolation.c	mm/page_isolation: do not isolate the max order page	2020-12-15 12:13:45 -08:00
page_owner.c	mm: page_owner: detect page_owner recursion via task_struct	2021-04-30 11:20:36 -07:00
page_poison.c	mm: page_poison: print page info when corruption is caught	2021-04-30 11:20:36 -07:00
page_reporting.c	mm/page_reporting: use list_entry_is_head() in page_reporting_cycle()	2021-02-24 13:38:30 -08:00
page_reporting.h
page_vma_mapped.c	mm/page_vma_mapped.c: add colon to fix kernel-doc markups error for check_pte	2020-12-15 12:13:41 -08:00
page-writeback.c	mm: page-writeback: simplify memcg handling in test_clear_page_writeback()	2021-04-30 11:20:37 -07:00
pagewalk.c
percpu-internal.h	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
percpu-km.c
percpu-stats.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
percpu-vm.c
percpu.c	percpu: make pcpu_nr_empty_pop_pages per chunk type	2021-04-09 13:58:38 +00:00
pgalloc-track.h
pgtable-generic.c	mm/pgtable-generic.c: optimize the VM_BUG_ON condition in pmdp_huge_clear_flush()	2021-02-24 13:38:30 -08:00
process_vm_access.c	mm/process_vm_access.c: include compat.h	2021-01-12 18:12:54 -08:00
ptdump.c	mm: ptdump: fix build failure	2021-04-16 16:10:37 -07:00
readahead.c	mm: Implement readahead_control pageset expansion	2021-04-23 10:14:29 +01:00
rmap.c	mm/rmap: correct obsolete comment of page_get_anon_vma()	2021-02-26 09:41:01 -08:00
rodata_test.c	mm/rodata_test.c: fix missing function declaration	2020-08-21 09:52:53 -07:00
shmem.c	shmem: allow reporting fanotify events with file handles on tmpfs	2021-04-19 16:03:48 +02:00
shuffle.c	mm: eliminate "expecting prototype" kernel-doc warnings	2021-04-16 16:10:36 -07:00
shuffle.h
slab_common.c	mm/slab_common: provide "slab_merge" option for !IS_ENABLED(CONFIG_SLAB_MERGE_DEFAULT) builds	2021-04-30 11:20:36 -07:00
slab.c	Merge branch 'for-mingo-rcu' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu into core/rcu	2021-04-11 14:31:43 +02:00
slab.h	RCU changes for this cycle were:	2021-04-28 12:00:13 -07:00
slob.c	mm: Don't build mm_dump_obj() on CONFIG_PRINTK=n kernels	2021-03-08 14:18:46 -08:00
slub.c	mm/slub.c: trivial typo fixes	2021-04-30 11:20:36 -07:00
sparse-vmemmap.c
sparse.c	mm/memory_hotplug: guard more declarations by CONFIG_MEMORY_HOTPLUG	2020-10-16 11:11:18 -07:00
swap_cgroup.c
swap_slots.c	mm/swap_slots.c: remove redundant NULL check	2021-02-24 13:38:28 -08:00
swap_state.c	mm: add FGP_ENTRY	2021-02-26 09:40:59 -08:00
swap.c	mm: remove pagevec_lookup_entries	2021-02-26 09:40:59 -08:00
swapfile.c	swap: fix swapfile read/write offset	2021-03-02 17:25:46 -07:00
truncate.c	mm: remove pagevec_lookup_entries	2021-02-26 09:40:59 -08:00
usercopy.c
userfaultfd.c
util.c	mm: move page_mapping_file to pagemap.h	2021-04-30 11:20:37 -07:00
vmacache.c
vmalloc.c	mm: Don't build mm_dump_obj() on CONFIG_PRINTK=n kernels	2021-03-08 14:18:46 -08:00
vmpressure.c
vmscan.c	mm/vmscan: restore zone_reclaim_mode ABI	2021-02-24 13:38:34 -08:00
vmstat.c	mm/vmstat.c: erase latency in vmstat_shepherd	2021-02-26 09:41:00 -08:00
workingset.c	mm: workingset: clarify eviction order and distance calculation	2021-02-24 13:38:34 -08:00
z3fold.c	z3fold: prevent reclaim/free race for headless pages	2021-03-25 09:22:55 -07:00
zbud.c	mm: set the sleep_mapped to true for zbud and z3fold	2021-02-26 09:41:01 -08:00
zpool.c	mm/zswap: add the flag can_sleep_mapped	2021-02-26 09:41:01 -08:00
zsmalloc.c	mm/zsmalloc.c: use page_private() to access page->private	2021-02-26 09:41:01 -08:00
zswap.c	mm/zswap: add the flag can_sleep_mapped	2021-02-26 09:41:01 -08:00